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Preface 


The  National  Defense  Authorization  Act  for  Fiscal  Year  2014  directed  the  Secretary  of  Defense 
to  assess  “the  feasibility  of  including  a  360-degree  assessment  approach  ...  as  part  of  perfor¬ 
mance  evaluation  reports”  and  to  report  to  Congress  on  the  findings  of  that  assessment.  The 
Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness  (OUSD/P&R)  asked 
the  RAND  Corporation  to  assist  with  this  assessment,  and  this  report  provides  OUSD/P&R 
with  information  relevant  to  that  request.  OUSD/P&R  asked  RAND  not  only  to  explore  the 
use  of  360-degree  assessments  (360s)  in  the  military  for  evaluation  purposes,  as  requested  by 
Congress,  but  also  to  provide  information  on  the  role  of  360s  more  broadly.  Thus,  our  study 
explores  the  pros  and  cons  of  using  360s  for  evaluation  and  development  purposes  in  the 
military. 

Our  research  is  based  on  information  gleaned  from  a  number  of  sources:  existing  research 
literature  and  expert  guidance  on  360  best  practices;  policy  documents  and  other  sources  sum¬ 
marizing  current  performance  and  promotion  practices  in  the  military  services,  including  the 
use  of  360s;  and  interviews  with  a  sample  of  stakeholders  and  subject-matter  experts  in  the 
Department  of  Defense.  The  results  of  our  research  suggest  that  using  360  feedback  as  part  of 
the  military  performance  evaluation  system  is  not  advisable  at  this  time,  though  the  services 
could  benefit  from  using  360s  as  a  tool  for  leader  development  and  to  gain  an  aggregate  view 
of  leadership  across  the  force. 

The  research  reported  here  will  be  of  interest  to  individuals  in  the  services,  Congress,  and 
the  Department  of  Defense  who  are  responsible  for  military  personnel  policies,  or  who  are 
considering  implementing  360  feedback  with  military  personnel. 

This  research  was  sponsored  by  the  Office  of  the  Under  Secretary  of  Defense  for  Per¬ 
sonnel  and  Readiness  and  conducted  within  the  Forces  and  Resources  Policy  Center  of  the 
RAND  National  Defense  Research  Institute,  a  federally  funded  research  and  development 
center  sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff,  the  Unified  Com¬ 
batant  Commands,  the  Navy,  the  Marine  Corps,  the  defense  agencies,  and  the  defense  Intel¬ 
ligence  Community. 

For  more  information  on  the  RAND  Forces  and  Resources  Policy  Center,  see  http:// 
www.rand.org/nsrd/ndri/centers/frp.html  or  contact  the  director  (contact  information  is  pro¬ 
vided  on  the  web  page). 


Contents 


Preface . iii 

Figures  and  Tables . vii 

Summary .  ix 

Acknowledgments . xiii 

CHAPTER  ONE 

Introduction  .  1 

Research  Questions  . 2 

Method . 2 

Organization  of  This  Report . 5 

CHAPTER  TWO 

What  Are  360-Degree  Assessments,  and  How  Are  They  Different  from  Other  Assessments?  ...  7 

How  Organizations  Use  360-Feedback  Programs . 9 

Employee  Views  on  360  Feedback . 10 

Evidence  of  Performance  Improvement  Using  360s . 11 

Designing  and  Implementing  a  360-Feedback  System . 12 

CHAPTER  THREE 

Performance  Evaluation  and  Promotion  Processes  in  the  Military  Services . 17 

Performance  Evaluation  Systems . 17 

Promotion  Process .  20 

Summary .  22 

CHAPTER  FOUR 

360-Feedback  Systems  Currently  in  Use  in  the  Military .  23 

The  Army’s  Multi-Source  Assessment  and  Feedback  Program . 25 

Other  360  Efforts  in  the  Army .  28 

Related  Efforts  . 29 

CHAPTER  FIVE 

Using  360  Feedback:  Evaluation  Versus  Development . 33 

DoD  Perspectives  on  the  Use  of  360s  for  Evaluation . 33 

DoD  Perspectives  on  the  Use  of  360s  for  Development .  38 

Overall  Conclusions  on  Advisability  of  Evaluation  Versus  Development . 39 


v 


vi  360-Degree  Assessments:  Are  They  the  Right  Tool  for  the  U.S.  Military? 


CHAPTER  SIX 

Using  360  Feedback:  Other  Implementation  Issues .  43 

Identifying  the  Right  Raters .  43 

Incorporating  Coaching  and  Follow-Up . 45 

Addressing  Fairness  and  Legal  Concerns .  46 

Maintaining  Trust  Within  the  Organization . 47 

Creating  a  Supportive  Organizational  Culture . 49 

Obtaining  Accurate  Ratings .  50 

Flarming  the  Organization  When  Using  360s  for  Evaluation . 53 

Making  360s  Logistically  Feasible  and  Conserving  Resources . 55 

Including  Better  Measures  of  Success . 57 

Tailoring  360s  to  the  Services’  Needs . 59 

CHAPTER  SEVEN 

Implications  for  the  Military  Services . 61 

Purpose  and  Design  Matter . 61 

Recommendations  for  Using  360s  in  the  Military . 62 

Abbreviations . 67 

References . 69 


Figures  and  Tables 


Figures 

2.1.  Illustration  of  360-Degree  Assessment . 8 

2.2.  How  Companies  Use  360-Degree  Assessment  Results . 9 

Tables 

1.1.  Interview  Participants . 4 

2.1.  Examples  of  How  360  Success  Could  Be  Defined  from  Various  Viewpoints . 12 

2.2.  Summary  of  Company  Experiences  Using  360  Feedback . 13 

3.1.  Comparison  of  the  Military  Services’  Performance  Evaluation  Systems . 19 

3.2.  0-6  and  Below  Performance  Evaluation  Criteria .  20 

3.3.  General/Flag  Officer  Performance  Evaluation  Criteria . 21 

4.1.  Examples  of  Existing  360  Efforts  in  Each  Service .  24 

5.1.  Comments  About  Using  360s  for  Evaluation  Purposes .  34 

5.2.  Negative  Comments  About  Evaluative  Use,  by  Service  Group . 35 

5.3.  Comments  About  Using  360s  for  Developmental  Purposes .  38 

5.4.  Number  of  Interview  Groups  That  Believe  It  Is  Advisable  to  Use  360s  for 

Evaluation  Versus  Development . 39 


vii 


Summary 


360-degree  assessments  (360s) — in  which  employees  receive  feedback  from  supervisors,  subor¬ 
dinates,  and  peers — have  become  popular  personnel  management  tools.  The  most  common  use 
of  360s  is  as  a  feedback  and  development  tool,  and  many  civilian  organizations  have  embraced 
360s  as  one  of  several  ways  to  develop  their  workforce.  Likewise,  the  military  has  also  begun 
exploring  the  use  of  360s  as  a  way  to  contribute  to  personnel  development,  most  commonly  for 
officers  in  leadership  positions. 

In  spite  of  their  popularity,  however,  360s  are  not  a  panacea.  Even  when  used  for  devel¬ 
opment  purposes,  the  use  of  360s  can  face  legitimate  criticism.  Given  that  360s  can  be  time- 
consuming  to  complete  and  their  effectiveness  can  vary,  some  worry  that  their  use  could  be 
a  significant  waste  of  resources.  Moreover,  experts  also  caution  that,  when  used  in  a  way  in 
which  the  results  will  ultimately  impact  high-stakes  career  outcomes  (such  as  promotions), 
360s  can  lead  to  distrust  and  have  the  potential  to  do  more  harm  than  good.  Although  many 
of  these  arguments  against  using  360s  are  legitimate,  some  could  be  mitigated  with  careful 
attention  to  the  design  of  the  360.  Others,  however,  should  lead  users  to  think  carefully  about 
when  it  is  worthwhile  to  implement  360s  and  when  it  is  not. 

To  better  understand  these  criticisms  and  other  pros  and  cons  of  using  360s  in  a  military 
context,  the  Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness  (OUSD/ 
P&R)  reached  out  to  the  RAND  Corporation  for  assistance.  Part  of  the  impetus  for  OUSD/ 
P&R’s  request  was  the  need  to  respond  to  section  571  of  the  National  Defense  Authorization 
Act  (NDAA)  for  Fiscal  Year  2014,  which  required  the  Secretary  of  Defense  to 

submit  to  the  Committees  on  Armed  Services  of  the  Senate  and  the  House  of  Representa¬ 
tives  a  report  containing  the  results  of  an  assessment  of  the  feasibility  of  including  a  360- 
degree  assessment  approach,  modeled  after  the  current  Department  of  the  Army  Multi- 
Source  Assessment  and  Feedback  (MSAF)  Program,  as  part  of  performance  evaluation 
reports. 

This  report  provides  OUSD/P&R  with  information  relevant  to  that  request.  OUSD/P&R 
asked  RAND  to  explore  the  use  of  360s  in  the  military  for  evaluation  purposes,  as  instructed 
in  the  NDAA,  but  also  asked  RAND  for  assistance  in  understanding  the  role  of  360  feed¬ 
back  more  broadly.  As  a  result,  this  study  explores  not  only  the  pros  and  cons  of  using  360s 
for  evaluative  purposes  in  the  military  but  also  the  pros  and  cons  of  its  use  for  developmental 
purposes. 


IX 


x  360-Degree  Assessments:  Are  They  the  Right  Tool  for  the  U.S.  Military? 


Research  Questions  and  Approach 

We  developed  four  overarching  questions  to  help  guide  the  department’s  understanding  of 
major  policy  issues  and  impacts  when  implementing  360s  in  the  services: 

•  What  is  known  about  360-degree  assessments? 

•  Are  the  military  services  already  using  360s?  If  so,  how  are  they  using  them? 

•  Would  it  be  advisable  to  implement  360s  for  development  or  evaluation  purposes  for  all 
officers1  in  the  military?  Why  or  why  not? 

•  What  implementation  challenges  should  the  services  be  aware  of  to  ensure  success? 

The  information  supporting  our  analysis  is  drawn  from  several  sources:  (1)  existing 
research  on  organizational  experience  with  360s;  (2)  documentation,  policies,  and  statutes  on 
promotions  and  performance  evaluations  in  the  military  services;  and  (3)  structured  interviews 
with  personnel  knowledgeable  about  the  largest  360s  currently  in  use  in  the  services  and  with 
key  stakeholders  and  military  personnel  groups  from  across  the  services,  the  Joint  Staff,  and 
the  Office  of  the  Secretary  of  Defense.  Each  of  these  sources  provides  a  unique  but  relevant 
contribution  to  the  report. 


The  Great  360  Debate 

The  debate  about  360s  centers  on  how  360  feedback  is  used — specifically,  whether  it  should 
be  used  for  developmental  purposes,  for  evaluation  purposes,  or  both.  Research  on  360  feed¬ 
back  programs  suggest  that  the  vast  majority  of  companies  that  implement  360s  use  them 
for  development-related  purposes,  such  as  training,  annual  development  goals,  and  succession 
planning.  Few  use  360  results  solely  for  evaluation  purposes  that  affect  personnel  decisions, 
such  as  promotions  and  salary  increases. 

To  the  degree  that  the  military  currently  uses  360s,  the  purpose  is  entirely  for  individual 
development.  Each  of  the  military  services  and  the  Joint  Staff  uses  some  version  of  360-degree 
assessments.  The  Army,  with  the  most  experience  by  far,  has  spent  14  years  developing  its 
Multi-Source  Assessment  and  Feedback  program,  which  now  has  widespread  implementation. 
In  the  other  services,  the  use  of  360s  is  more  targeted — generally  directed  at  senior  leadership 
or  toward  high-potential  officers  as  part  of  the  military  education  system.  A  number  of  the 
programs  are  in  early  stages  of  development  and  are  undergoing  pilot  tests  on  small  subsets  of 
the  population.  All  of  the  programs  are  developmental  in  nature,  and  their  small  size  allows  for 
personalized  coaching  support  in  the  interpretation  and  utilization  of  results.  To  date,  none  of 
the  services  uses  360s  as  an  evaluation  tool  in  the  performance  appraisal  process. 

Should  the  military  services  consider  broader  use  and  implement  360s  as  part  of  perfor¬ 
mance  evaluation?  Should  the  use  of  360s  be  mandated,  even  if  only  for  developmental  pur¬ 
poses?  Are  the  military  services  on  track?  Our  discussions  with  stakeholders  throughout  the 
Department  of  Defense  resulted  in  a  fairly  consistent  response  to  these  questions. 


1  The  Office  of  the  Secretary  of  Defense  suggested  that  we  focus  on  active-duty  officers  only,  with  the  acknowledgment 
that  the  question  of  360  uses  could  certainly  be  relevant  to  other  military  populations  as  well  (civilians,  enlisted,  reserve, 
and  guard). 


Summary  xi 


"No"  for  Evaluation 

Based  on  our  research  both  within  and  outside  the  military  setting,  we  advise  against 
incorporating  360s  in  the  officer  evaluation  system  at  this  time.  Although  it  might  be 
technically  feasible,  we  advise  against  it  for  several  reasons.  For  one,  it  could  ruin  develop¬ 
mental  applications,  and  the  Army’s  experience  shows  how  long  it  can  take  to  develop  trust 
in  such  a  process.  Trying  to  implement  two  systems  could  be  confusing  to  raters  and  increase 
survey  burden  on  the  force.  Because  of  the  complexity  of  using  360s  in  the  evaluation  system, 
mandating  their  use  could  lead  to  largely  useless  or  meaningless  ratings  that  are  difficult  to 
interpret  without  considerable  context.  Perhaps  the  key  concern  in  using  360s  beyond  develop¬ 
mental  purposes  is  their  impact  on  selection  boards  and  the  promotion  process.  Under  current 
law,  if  360s  are  included  in  an  officer’s  formal  evaluation  report,  the  results  of  that  assessment 
would  be  part  of  the  documentation  provided  to  selection  boards.  That  said,  the  services  could 
consider  other  alternatives  for  incorporating  a  broader  range  of  perspectives,  including  from 
peers  and  subordinates,  into  the  performance  evaluation  system — though  identifying  specific 
alternatives  is  beyond  the  scope  of  this  study. 

"Yes"  for  Development,  but  Proceed  with  Care 

We  do  advise  the  use  of  360s  for  development  purposes  for  people  at  higher  grades  or 
in  leadership  positions.  We  target  leadership  positions  because  360s  require  a  significant 
investment  in  time  from  service  members  and  from  raters,  and  the  benefits  of  360  feedback  for 
those  in  leadership  positions  are  most  obvious.  The  tool  could  be  made  available  as  a  service 
to  individuals  hoping  to  improve,  along  with  coaching  to  help  service  members  evaluate  the 
results  and  incorporate  them  into  self-improvement  goals.  360s  could  also  be  used  to  provide 
an  aggregate  view  of  leadership  performance  across  the  force — something  that  current  tools 
are  not  necessarily  well  positioned  to  provide.  Leaders  could  identify  locations  with  leadership 
issues  and  could  use  360  results  to  identify  force-wide  strengths  and  weaknesses.  The  bottom 
line  is  that  making  360  feedback  available  for  developmental  use  in  the  military  services  is  a 
good  idea  and  is  essentially  how  the  tool  is  being  used  today.  Each  of  the  services  uses  360 
assessments  in  ways  that  support  their  personnel  management  goals  and  are  in  line  with  service 
culture. 

Current  Implementation  Is  on  the  Right  Track 

Overall,  our  interviews  showed  that  the  spirit  of  360  clearly  resonates  with  the  services. 

The  services  value  good  leadership  behaviors  and  tools  that  can  help  develop  good  leaders. 
360s  are  one  tool  that  has  value  in  developing  leaders.  Despite  their  utility,  however,  360s  are 
not  a  silver  bullet.  Experts  in  the  field  of  personnel  research  agree  and  caution  that  overzealous 
use  of  360s  without  careful  attention  to  content,  design,  and  delivery  can  be  harmful  to  an 
organization  and  not  worth  the  time  and  money  spent  on  them.  Thus,  mandating  360  assess¬ 
ments  force-wide,  even  for  development  purposes,  is  not  necessarily  the  right  answer  to  solving 
leadership  problems  within  the  services  and  could  waste  significant  resources.  Rather  it  is  more 
advisable  to  allow  the  services  to  continue  on  their  current  paths,  expanding  the  use  of  360s  in 
a  way  that  is  tailored  to  individual  service  needs  and  goals. 
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CHAPTER  ONE 


Introduction 


360-degree  assessments  (360s) — in  which  employees  receive  feedback  from  supervisors,  sub¬ 
ordinates,  and  peers — have  grown  to  be  popular  tools  in  the  private  sector.  The  most  common 
use  of  the  360  is  as  a  feedback  and  development  tool.  Because  performance  improvement 
requires  feedback  about  one’s  strengths  and  weaknesses,  offering  insights  into  how  subordi¬ 
nates,  peers,  and  others  view  an  individual  could  be  a  useful  and  powerful  tool  for  employee 
development.  As  a  result,  many  civilian  organizations  have  embraced  360s  as  one  of  several 
ways  to  develop  their  workforce. 

For  many  of  the  same  reasons,  the  military  has  also  begun  exploring  the  use  of  360s  as 
a  way  to  contribute  to  service  member  development,  particularly  officers  in  leadership  posi¬ 
tions.  For  example,  the  Army  currently  uses  the  Army  Multi-Source  Assessment  and  Feedback 
(MSAF)  as  a  mandatory  developmental  tool  for  all  of  its  officers  and  as  an  optional  develop¬ 
mental  tool  for  its  civilian  and  enlisted  personnel.  That  effort,  which  started  as  a  small  pilot  in 
the  early  2000s,  has  grown  over  the  past  14  years  to  the  large-scale  effort  in  place  today.  Other 
military  services  and  the  Joint  Staff  are  exploring  the  use  of 360s  for  similar  purposes,  although 
some  of  their  implementations  are  still  largely  in  their  infancy  and  most  are  on  a  much  smaller 
scale. 

In  spite  of  their  popularity,  however,  360s  are  not  a  panacea.  Even  when  used  for  devel¬ 
opment  purposes,  they  can  face  legitimate  criticism.  Because  360s  can  be  time  consuming  to 
complete  and  their  effectiveness  can  vary,  some  worry  that  the  use  of  360s  could  be  a  signifi¬ 
cant  waste  of  resources.  Moreover,  experts  also  caution  that,  when  used  in  a  way  in  which  the 
results  will  ultimately  impact  high-stakes  career  outcomes  (such  as  promotions),  they  can  lead 
to  distrust  and  have  the  potential  to  do  more  harm  than  good.  Although  many  of  these  argu¬ 
ments  against  using  360s  are  legitimate,  some  could  be  mitigated  with  careful  attention  to 
how  the  tool  is  designed.  Others,  however,  should  lead  users  to  think  carefully  about  when  it 
is  worthwhile  to  implement  it  and  when  it  is  not. 

To  better  understand  these  criticisms  and  other  pros  and  cons  of  using  360s  in  a  military 
context,  the  Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness  (OUSD/ 
P&R)  reached  out  to  the  RAND  Corporation  for  assistance.  Part  of  the  impetus  for  this 
request  was  the  need  to  respond  to  section  571  of  the  National  Defense  Authorization  Act 
(NDAA)  for  Fiscal  Year  2014,  which  required  the  Secretary  of  Defense  to 

submit  to  the  Committees  on  Armed  Services  of  the  Senate  and  the  Flouse  of  Representa¬ 
tives  a  report  containing  the  results  of  an  assessment  of  the  feasibility  of  including  a  360- 
degree  assessment  approach,  modeled  after  the  current  Department  of  the  Army  Multi- 
Source  Assessment  and  Feedback  (MSAF)  Program,  as  part  of  performance  evaluation 
reports. 
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This  report  is  therefore  designed  to  provide  OUSD/P&R  with  information  relevant  to 
that  request.  OUSD/P&R  asked  RAND  to  explore  the  use  of  the  360s  in  the  military  for 
evaluation  purposes  specifically  because  the  NDAA  requested  it,  and  it  asked  RAND  to  provide 
greater  detail  on  the  existing  MSAF  program  in  response  to  its  specific  mention  in  the  NDAA. 

In  addition,  OUSD/P&R  asked  RAND  for  assistance  in  understanding  the  role  of  360 
feedback  more  broadly.  As  a  result,  we  expanded  the  goal  of  the  present  research  effort  to  also 
address  whether  360s  would  be  a  useful  tool  for  the  military  for  developmental  purposes.  Our 
study  was  therefore  designed  to  explore  not  only  the  pros  and  cons  of  using  360s  for  evalua¬ 
tive  purposes  in  the  military  but  also  the  pros  and  cons  of  its  use  for  developmental  purposes. 


Research  Questions 

There  are  many  questions  germane  to  the  feasibility  and  potential  success  of  using  360s  for 
military  personnel.  However,  answers  necessarily  depend  on  the  stated  purpose  and  the  focal 
population  for  the  360.  As  a  result,  answering  every  question  relevant  to  implementing  360s 
for  all  possible  uses  in  the  military  was  far  beyond  the  scope  of  this  study.  Instead,  we  focused 
our  questions  on  broad  topics  intended  to  help  guide  the  Office  of  the  Secretary  of  Defense’s 
(OSD)  understanding  of  the  major  policy  issues  and  impacts  when  implementing  360s  in  the 
services. 

Because  the  NDAA  language  seemed  to  imply  a  force-wide  implementation  of  360s,  we 
consulted  with  OUSD/P&R  to  determine  how  best  to  scope  the  population  of  interest  for  our 
research  questions.  Based  on  that  guidance,  we  focused  our  efforts  on  active-duty  officers  only, 
acknowledging  that  the  use  of  360s  could  be  relevant  to  other  military  populations  as  well 
(civilians,  enlisted,  reserve,  and  guard). 

In  consultation  with  our  sponsor,  we  identified  the  following  broad  questions  as  the  pri¬ 
mary  focus  of  our  efforts: 

•  What  is  known  about  360-degree  assessments? 

•  Are  the  military  services  already  using  360s?  If  so,  how  are  they  using  them? 

•  Would  it  be  advisable  to  implement  360s  for  developmental  or  evaluation  purposes  for  all 
officers  in  the  military?  Why  or  why  not? 

•  What  implementation  challenges  should  the  services  be  aware  of  to  ensure  success? 


Method 

Our  analysis  pulled  information  from  several  sources  to  address  these  questions.  We  conducted 
structured  interviews  with  developers  of  360s  currently  in  use  in  the  services  and  with  key 
stakeholders  and  military  personnel  groups.  We  drew  on  existing  research  and  expert  advice  on 
organizational  experience  with  360-degree  assessment,  examining  such  issues  as  how  experts 
recommend  using  this  tool,  reactions  to  360  feedback,  the  accuracy  of  360  feedback,  and 
effects  of  360  feedback  on  subsequent  performance.  Finally,  we  reviewed  documentation,  poli¬ 
cies,  and  statutes  on  promotions  and  performance  evaluations  in  the  military  services  to  learn 
about  the  military’s  current  use  of  360s  and  how  360s  would  complement  or  duplicate  exist¬ 
ing  processes.  Each  of  these  sources  provides  a  unique  but  relevant  contribution  to  this  report. 
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In  the  sections  that  follow,  we  pull  from  these  sources  in  responding  to  the  questions  posed, 
though  not  all  sources  are  relevant  for  addressing  every  topic  area. 

Stakeholders  and  Subject-Matter  Expert  Interviews 

The  purpose  of  our  interviews  was  to  collect  input  from  the  policymakers  involved  in  personnel 
evaluation  or  administration  of  existing  360s.  Toward  that  end,  the  project  sponsor  identified 
13  individuals  for  us  to  interview  from  across  the  services,  the  Joint  Staff,  and  OSD.  Though 
limited  in  number,  the  individuals  interviewed  are  senior  policy  stakeholders  and  therefore 
contribute  expert  opinion  to  the  evaluation,  assessment,  promotion,  or  development  process. 
In  addition  to  interviewing  ten  of  the  original  13  individuals,  we  conducted  eight  interviews 
with  other  subject-matter  experts  recommended  by  the  original  13,  who  could  help  shed  light 
on  specific  implementation  challenges  and  benefits. 

In  total,  we  conducted  interviews  with  19  distinct  groups  during  the  project  time  frame, 
as  shown  in  Table  1.1.  Each  interview  included  the  primary  contact  and  often  one  or  more  col¬ 
leagues  invited  by  the  primary  contact. 

Interviews  were  semistructured  and  covered  the  following  questions: 

•  What  types  of  360-degree  assessment  activities  is  your  service  or  organization  currently 
using?  Are  they  for  development  or  evaluation  purposes? 

•  If  the  military  were  to  adopt  360-degree  assessment  for  evaluation  purposes,  what  do  you 
foresee  as  the  possible  positive  and  negative  consequences? 

•  How  about  for  development  purposes? 

•  How  feasible  would  it  be  to  implement  this  for  all  officers?  Are  there  logistical  concerns 
(stakeholder  issues,  technology  issues,  policy  and  legal  impacts,  financial  costs  or  other 
resource  issues,  etc.)? 

•  How  advisable  would  it  be  for  the  military  to  implement  360-degree  assessment  for  devel¬ 
opment  and/or  evaluation? 

•  What  else  should  we  consider? 

However,  we  considered  the  discussions  exploratory  and  therefore  probed  with  additional 
related  questions  as  relevant. 

We  analyzed  interview  comments  in  two  ways.  First,  we  identified  a  series  of  comments 
raised  frequently  during  the  discussions  and  provided  counts  of  the  frequency  to  highlight  the 
most  prominent  concerns  of  the  group.1  However,  because  each  participant  brought  unique 
perspectives,  experiences,  and  insights  to  the  study,  there  was  also  value  in  conveying  a  com¬ 
prehensive  picture  of  all  of  the  ideas  and  thoughts  that  were  raised,  regardless  of  the  frequency. 
These  less  frequent  comments  were  particularly  insightful  in  that  they  identified  issues  that 
appear  to  be  legitimate  concerns  that  others  had  perhaps  not  previously  considered.  We  there¬ 
fore  summarize  their  comments  and  offer  paraphrased  quotes  from  our  notes  to  illustrate  a 
number  of  these  points  throughout  the  report. 

Although  we  designed  our  interviews  to  focus  on  policymakers  with  influence  over  or 
responsibility  for  360  policies  in  the  services,  we  acknowledge  that  they  are  not  the  only  ser¬ 
vice  members  and  military  leaders  who  might  have  opinions  about  360s.  For  example,  the 


1  Three  RAND  researchers  coded  the  interview  notes  and  iteratively  modified  the  major  categories  and  subcategories  to 
reach  agreement  on  a  final  set  of  coding  categories. 
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Table  1.1 

Interview  Participants 


Group 

Name  and  Affiliation 

OSD 

RADM  Margaret  D.  (Peg)  Klein,  Senior  Advisor  to  the  Secretary  of  Defense 

Lt  Col  Kevin  Basik  (U.S.  Air  Force  representative) 

COL  Robert  Taradash  (U.S.  Army  representative) 

LtCol.  Darrell  Platz  (U.S.  Marine  Corps  representative) 

Steven  Strong  and  James  Schwenk,  Office  of  the  General  Counsel 

Joint  Staff 

Leslie  A.  Purser,  Joint  Education  Advisor,  J7  Division 

Lt  Col  Paul  Amrhein 

BG  Margaret  W.  Burcham,  Director  for  Manpower  and  Personnel,  J1 

LtGen  Thomas  Waldhauser,  Director  for  Joint  Force  Development,  J7 

COL  Richard  M.  (Mike)  Cabrey,  Chief  of  Staff  to  the  Director,  Joint  Force  Development,  J7 

Jerome  M.  Lynes,  Deputy  Director  for  Joint  Education  and  Doctrine,  Joint  Chiefs  of  Staff 

Army 

COL  Christopher  D.  Croft,  Director,  Center  for  Army  Leadership 

Dr.  Jon  Fallesen 

Anthony  J.  Gasbarre 

Dr.  Melissa  Wolfe 

Clark  Delavan 

LTC  Chris  Taylor 

LTC  Johnny  Oliver  (representing  the  Office  of  MG  Thomas  Seamands,  Director  of  Military 
Personnel  Management,  Army  G-1) 

COL  Robert  Bennett 

COL  Michael  E.  Masley 

Albert  Eggerton 

COL  Joseph  R.  Callaway,  General  Officer  Management  Office 

Anthony  J.  Stamilio,  Deputy  Assistant  Secretary  of  the  Army 

Air  Force 

Brig  Gen  Brian  T.  Kelly,  Director,  Force  Management  Policy 

Jeffrey  Mayo,  Deputy  Assistant  Secretary  of  the  Air  Force  for  Force  Management  Integration 

Russell  J.  Frasz,  Director,  Force  Development 

Col  Christopher  E.  Craige,  General  Officer  Management  Office 

Marine  Corps  Col.  Scott  Erdelatz,  Director,  Lejeune  Leadership  Institute 


Navy 

RADM  Frederick  Roegge,  Director,  Military  Personnel  Plans  and  Policy  (N13) 

CAPT  Jeffrey  Krusling 

CDR  Nicole  DeRamus-Suazo,  Ph.D.,  Deputy  Director  for  Navy  Flag  Officer  Development 

N/A 

LTG  Walt  Ulmer  (Ret.),  former  Director,  Center  for  Creative  Leadership 

broader  force  may  strongly  desire  opportunities  to  provide  upward  feedback  and  may  believe 
that  doing  so  would  be  valuable.  Junior  officers  in  particular  might  want  such  an  avenue  to 
voice  anonymous  opinions.  Others  in  the  policy  community  also  have  views  on  360s.  In  their 
report  reviewing  the  2010  Quadrennial  Defense  Review,  Hadley  and  Perry  (2010)  recom¬ 
mended  that  Congress  mandate  360s  for  all  officers  in  the  services.  That  said,  the  interviewees 
who  were  invited  to  participate  provide  important  perspectives  from  policymakers  currently  in 
a  position  to  shape  future  efforts  to  implement  360s  on  a  larger  scale. 
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Research  and  Expert  Advice  on  360s 

The  literature  on  360s  is  expansive.  Hundreds  of  journal  articles  and  dozens  of  books  provide 
expert  guidance  on  360  best  practices,  most  published  in  the  1990s  and  2000s.  They  debate 
the  pros  and  cons  of  using  360s,  discuss  the  impacts  of  360s  on  workplace  performance,  and 
address  issues  of  when  and  how  to  design  360s.  Because  a  number  of  excellent  comprehensive 
reviews  of  the  existing  research  can  be  found  elsewhere  (see,  for  example,  The  Handbook  of  Multi¬ 
source  Feedback  by  Bracken,  Timmreck,  and  Church,  2001),  and  because  the  literature  itself  is 
vast,  our  intention  was  not  to  provide  yet  another  comprehensive  review.  Instead,  we  sought 
to  use  the  literature  to  accomplish  two  goals.  First  we  aimed  to  provide  a  quick  summary  of 
360s  as  general  background  information  for  our  readers.  Second,  and  more  importantly,  where 
relevant,  we  pulled  in  existing  research  and  expert  advice  to  help  support  or  refute  some  of  the 
issues  raised  in  our  interviews  with  the  key  stakeholders  and  military  representatives. 

Our  review  therefore  focused  solely  on  findings  that  could  inform  the  decision  to  adopt 
a  360-feedback  program  and  factors  to  consider  in  program  design  and  implementation.  We 
intentionally  omitted  some  heavily  researched  areas  of  360s  that  may  be  less  relevant  to  policy¬ 
makers.  For  example,  there  is  an  abundant  body  of  literature  on  individual  difference  charac¬ 
teristics  associated  with  360  ratings.2  While  it  might  be  important  for  coaches  and  mentors  to 
be  aware  of  how  these  characteristics  can  influence  ratings  in  order  to  help  put  the  ratings  in 
context  or  to  help  maximize  the  usefulness  for  some  individuals,  many  of  the  findings  point  to 
biological  and  individual  characteristics  that  are  stable  and  that  an  organization  cannot  affect 
on  a  large  scale.  Instead,  we  focused  on  factors  that  organizations  are  in  a  position  to  influence 
and  that  are  relevant  for  establishing  policy,  such  as  how  360  programs  are  designed  and  how 
the  information  is  used. 


Organization  of  This  Report 

The  remainder  of  this  report  responds  to  the  aforementioned  research  questions.  In  Chapter 
Two,  we  provide  a  brief  overview  of  360s  as  background  for  our  readers.  Because  the  NDAA 
language  proposes  using  360s  for  evaluation  purposes,  Chapter  Three  summarizes  the  exist¬ 
ing  performance  evaluation  and  promotion  process  used  by  the  military  services  at  the  time  of 
this  study.  The  next  chapter,  Chapter  Four,  offers  an  overview  of  the  military’s  current  use  of 
360s  (in  use  or  in  development  at  the  time  of  the  study),  with  special  attention  to  the  Army’s 
MSAF.  Chapters  Five  and  Six  summarize  the  key  concerns  raised  by  stakeholders  regarding 
implementation  of  360s  and  discuss  existing  research  that  addresses  those  concerns.  The  final 
chapter  discusses  implications  of  the  study’s  findings  and  provides  recommendations  for  how 
360s  should  be  used  in  the  military  services. 


2  Fleenor  et  al.  (2010)  contains  a  comprehensive  summary  of  research  on  congruence  of  self  and  other  rating  and  review 
findings  from  numerous  studies  on  effects  of  biographical  characteristics,  personality  traits,  and  other  dispositional  charac¬ 
teristics,  as  well  as  job-relevant  experience.  Ostroff,  Atwater,  and  Feinberg  (2004)  also  reviewed  the  influence  on  ratings  of 
rater-ratee  similarities  or  differences  in  characteristics,  such  as  age,  gender,  and  race.  Others  have  studied  individual  char¬ 
acteristics,  such  as  personality  and  dispositional  traits,  on  acceptance  of  360  feedback  (Smither,  London,  and  Richmond, 
2005),  goal  commitment  following  360  feedback  (Bono  and  Colbert,  2005),  and  performance  improvement  following  360 
feedback  (e.g.,  Heslin  and  Latham,  2004;  Smither,  London,  and  Richmond,  2005). 


CHAPTER  TWO 


What  Are  360-Degree  Assessments,  and  How  Are  They  Different 
from  Other  Assessments? 


Put  simply,  360s  are  a  tool  that  can  be  used  to  measure  an  employee’s  performance.  Like  any 
other  tool  used  for  this  purpose  (such  as  a  multiple- choice  test,  simulation  activity,  or  supervi¬ 
sory  ratings),  the  method  itself  is  not  inherently  good  or  bad.  Instead,  the  desired  goal  for  the 
tool,  what  it  is  intended  to  measure,  how  well  it  is  designed,  and  how  it  is  used  can  all  impact 
its  usefulness. 

Supervisor  ratings  are  the  most  common  way  in  which  employee  performance  is  mea¬ 
sured.  Although  a  supervisor  may  solicit  input  from  others  in  making  his  or  her  ratings,  in 
most  cases,  only  the  direct  supervisor  determines  a  subordinate’s  rating.  Supervisor  ratings 
are  typically  collected  as  part  of  an  employee’s  annual  performance  evaluation  and  are  used  to 
determine  salary  actions  or  promotions,  to  officially  document  inadequate  performance,  and 
to  provide  employees  with  feedback  on  how  their  supervisors  think  they  are  performing.  In 
many  organizations,  the  annual  performance  review  includes  a  developmental  component  in 
which  supervisors  may  use  the  formal  evaluation  as  an  opportunity  to  provide  the  employee 
with  developmental  feedback  and  coaching. 

However,  in  many  jobs,  a  direct  supervisor  is  not  the  only  person  who  is  in  a  position  to 
comment  on  a  person’s  performance.  Moreover,  in  some  jobs,  the  direct  supervisor  may  not 
even  be  the  best  situated  to  provide  an  accurate  assessment  of  certain  aspects  of  performance. 
For  example,  a  supervisor  may  not  be  in  a  position  to  see  how  someone  interacts  with  the 
people  below  him  or  her  on  a  day-to-day  basis.  As  a  result,  many  experts  in  personnel  evalua¬ 
tion  have  noted  that  supervisor  ratings  can  sometimes  provide  incomplete  pictures  of  employee 
performance  (Hedge,  Borman,  and  Birkeland,  2001;  Lepsinger  and  Lucia,  2001). 1  The  360 — 
which  includes  self-ratings  and  feedback  from  peers,  subordinates,  and  others  (such  as  cus¬ 
tomers)  in  addition  to  the  supervisor — is  one  tool  designed  to  help  fill  in  some  of  the  gaps  in 
supervisor  ratings. 

Also  known  as  multisource  or  multirater  assessments ,  it  is  this  360-degree  view  of  an 
employee’s  performance  (illustrated  in  Figure  2.1)  that  has  led  to  its  most  widely  recognized 
name:  360-degree  assessment.  In  keeping  with  its  most  well-recognized  name,  in  this  report,  we 
refer  to  it  as  360-degree  assessment  or,  more  simply,  360.  There  are  many  variants  on  the  concept 
of  360-degree  feedback  that  differ  from  that  illustrated  in  the  figure.  Some  assessments  include 
only  supervisors  and  subordinates;  others  include  only  views  from  supervisors  and  custom¬ 
ers.  Although  these  variants  may  be  distinguished  from  360s  by  some  other  authors  (see,  for 


1  Many  studies  have  also  examined  possible  sources  of  unreliability  in  supervisory  ratings.  For  examples,  see  Landy  and 
Farr,  1980;  Murphy,  2008;  and  Judge  and  Ferris,  1993. 
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Figure  2.1 

Illustration  of  360-Degree  Assessment 
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example,  Foster  and  Law,  2006),  360s  in  this  report  (except  where  we  specify  otherwise)  refers 
to  assessments  that  include  any  source  in  addition  to  or  instead  of  supervisors. 

360s  most  commonly  measure  interpersonal  competencies  or  “soft  skills,”  such  as  lead¬ 
ership,  teamwork,  or  customer  service,  that  are  valued  aspects  of  performance  in  business 
settings.  Jobs  for  which  a  central  aspect  of  performance  involves  managing  project  teams, 
supervising  or  leading  others,  or  other  interactions  with  people  internal  or  external  to  the 
organization  are  usually  well  suited  to  360  measurements.  For  those  employees,  performance 
on  the  job  may  be  synonymous  with  leadership  performance  and  teamwork.  However,  360s 
are  not  well  suited  for  measuring  all  aspects  of  performance  in  all  jobs,  such  as  employees  who 
work  in  isolation  (assembly  line  factory  workers,  for  example). 

Arguments  exist  for  and  against  using  360s  in  different  contexts.  For  example,  propo¬ 
nents  of  the  360  argue  that  it  provides  a  more  accurate  picture  of  individual  performance  than 
traditional  feedback  systems  because  it  includes  multiple  perspectives  of  employees’  perfor¬ 
mance  (e.g.,  Borman,  1997;  Tornow,  1993).  This  is  particularly  useful  for  aspects  of  perfor¬ 
mance  that,  like  interpersonal  skills,  cannot  be  assessed  through  purely  objective  measures, 
such  as  the  number  of  items  sold  or  produced.  In  addition,  ratings  from  multiple  raters  can  be 
more  reliable  than  ratings  from  a  single  rater  and  therefore  may  be  more  valid.  360s  can  also 
reduce  rater  biases  and  random  error  (Mount  et  ah,  1998).  Proponents  of  360  feedback  also 
speak  to  its  value  in  increasing  communication  among  employees  and  fostering  an  increased 
sense  of  participation  and  involvement  in  the  organization  that  is  consistent  with  the  use  of 
teams  and  other  participative  management  strategies.  360s  also  can  promote  a  sense  of  fairness 
or  organizational  justice  (e.g.,  Edwards,  Ewen,  and  Vendantam,  2001). 

However,  there  also  are  costs  to  360s.  Such  programs  are  resource-intensive  to  design, 
implement,  and  maintain.  360  reviews  are  time-consuming  for  participants,  who  may  be 
reluctant  to  give  feedback  or  be  forthright  in  their  feedback,  particularly  if  they  are  concerned 
that  their  comments  can  be  identified.  Additionally,  critics  note  that  360s  have,  in  many  cases, 
been  shown  to  be  ineffective  at  improving  behavior  and,  in  some  circumstances,  can  even 
harm  performance. 
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How  Organizations  Use  360-Feedback  Programs 

A  recent  survey  of  360-feedback  program  managers  from  211  organizations2  in  the  United 
States  and  Canada  showed  that  the  use  of  360  feedback  varies,  but  the  vast  majority  of  those 
surveyed  use  it  for  development-related  purposes  (Figure  2.2).  About  one-third  use  360s  for 
self-directed  development,  another  third  use  it  for  development  planning  (such  activities  as 
identifying  training  and  development  needs,  annual  development  goals,  and  succession  plan¬ 
ning),  and  another  third  use  it  to  make  both  development  and  personnel  decisions.  Few  of  the 
companies  surveyed  use  360  results  solely  for  “personnel  decisions  such  as  pay,  promotion,  and 
hiring”  (3D  Group,  2013). 

Many  researchers  have  described  the  pros  and  cons  of  using  360s  for  evaluation  purposes 
versus  developmental  purposes,  with  most  arguing  that  they  are  best  used  for  development 
only  (see  London,  2001,  for  example).  The  notion  that  360s  provide  better  information  because 
peers  and  subordinates  are  often  in  a  better  position  than  supervisors  to  evaluate  some  behav¬ 
iors  is  common  to  both  sides  of  the  debate.  360s  can  enhance  self-development  and  provide  a 
sound  basis  for  decisions.  Arguments  in  favor  of  using  360  feedback  for  evaluative  decisions 
contend  that  organizations  should  take  advantage  of  all  possible  benefits  from  their  investment 
in  360  programs.  On  the  developmental  side,  proponents  argue  that  360s  enhance  self-aware¬ 
ness,  which  in  turn  contributes  to  behavior  change,  and  that  individuals  value  others’  opinions 
and  feel  responsible  to  respond  to  them.  When  ratings  are  anonymous,  which  is  typically  the 
practice  in  360s,  raters  are  more  likely  to  be  forthright  in  their  feedback. 

Experts  have  offered  numerous  reasons  explaining  why  using  360  feedback  for  evalu¬ 
ation  decisions  is  counterproductive.  For  example,  London  (2001)  notes  that  feedback  may 


Figure  2.2 

How  Companies  Use  360-Degree  Assessment  Results 


RAND  RR998-2.2 


2  The  results  of  the  3D  Group  study  appear  to  be  based  on  a  convenience  sample  of  organizations.  As  a  result,  the  findings 
are  not  necessarily  representative  of  all  organizations  in  the  United  States.  Nevertheless,  the  sample  does  include  a  large 
number  of  well-known  corporations.  At  a  minimum,  the  study  can  provide  useful  insights  into  the  ways  in  which  those 
organizations  use  and  view  360s. 
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be  inflated  because  raters  find  it  stressful  and  threatening  to  share  negative  comments,  raters 
may  have  a  vested  interest  in  the  ratee  receiving  a  favorable  result,  and  ratees  may  nominate 
raters  who  are  more  likely  to  provide  positive  comments.  Experts  also  note  that  anonymity  can 
enable  raters  to  sabotage  the  ratee  or  can  lead  the  ratee  to  believe  that  raters  intend  to  sabotage 
him  or  her,  which  in  turn  will  lead  the  ratee  to  discount  negative  feedback.  While  there  are  few 
studies  that  directly  compare  360-feedback  systems  used  for  development  versus  evaluation, 
most  experts  still  advise  against  using  them  for  evaluation. 


Employee  Views  on  360  Feedback 

There  is  evidence  that  employees  generally  have  favorable  reactions  to  360s.  For  example, 
responses  to  surveys  of  more  than  1,400  employees  in  multiple  companies  (Edwards,  Ewen, 
and  Vendantam,  2001)  revealed  that  over  75  percent  of  respondents  reported  that  the  program 
added  useful  information  and  improved  process  fairness,  felt  that  feedback  was  helpful,  and 
recommended  the  360  process.  Data  from  59  project  evaluations  conducted  in  the  late  1990s 
across  a  range  of  industries  show  even  more  favorable  reactions  to  360  programs  (Edwards, 
Ewen,  and  Vendantam,  2001). 3 

Participants’  reactions  to  360s  are  important  because  they  affect  ratees’  beliefs  about  the 
quality  of  the  feedback  and  their  willingness  to  use  it.  In  the  literature  on  performance  evalu¬ 
ation  more  generally,  studies  show  that  reactions,  such  as  satisfaction  with  feedback  and  per¬ 
ceived  feedback  accuracy,  influence  subsequent  performance  (Jawahar,  2010).  Research  on  360 
feedback  showed  that  individuals  who  received  less-favorable  feedback  had  more-negative  reac¬ 
tions  and  perceived  the  feedback  as  less  accurate;  those  who  had  more-negative  reactions  and 
found  the  feedback  less  accurate  reported  that  the  feedback  was  less  useful  (Brett  and  Atwater, 
2001). 4  (For  a  comprehensive  review  on  performance  feedback  in  the  workplace,  see  London, 
2003.) 

Studies  have  also  identified  a  number  of  factors  that  influence  reactions  to  360s,  and 
many  of  these  results  point  to  strategies  for  360-feedback  system  design  and  implementa¬ 
tion.  One  factor  is  the  purpose  of  the  feedback;  studies  have  shown  that  participants  prefer 
360s  for  development  rather  than  for  evaluation.  Participants  believe  that  positive  outcomes 
are  more  likely  and  negative  outcomes  are  less  likely  when  using  subordinate  and  peer  feed¬ 
back  for  development  as  opposed  to  evaluation  (e.g.,  Bettenhausen  and  Fedor,  1997;  Farh, 
Cannella,  and  Bedeian,  1991;  Fedor,  Bettenhausen,  and  Davis,  1999).  For  example,  Fedor, 


3  In  the  latter  study,  consisting  of  the  59  project  evaluations,  data  were  available  regarding  the  purpose  of  the  evaluation, 
i.e.,  for  evaluation  or  development.  The  authors  reported  responses  for  only  one  item — the  usefulness  of  the  results — and, 
contrary  to  expectations,  found  higher  agreement  rates  in  companies  using  the  appraisal  for  evaluative  rather  than  develop¬ 
mental  purposes.  However,  the  authors  also  note  that  this  result  could  have  occurred  by  chance. 

4  These  findings  should  not  be  interpreted  to  mean  that  raters  should  avoid  providing  unfavorable  feedback.  In  a  study  of 
leaders  in  an  elite  unit  of  the  U.S.  military,  Smither,  London,  and  Richmond  (2005)  found  that  individuals  who  received 
more  unfavorable  ratings  (and  who  were  perceived  by  a  psychologist  as  more  defensive)  set  more  goals  for  improvement 
six  months  later  than  individuals  who  received  more  positive  ratings  (and  who  reported  the  feedback  as  more  valuable).  In 
addition,  in  a  study  of  performance  evaluation,  Jawahar  (2010)  found  that  characteristics  of  the  rater  and  the  way  in  which 
feedback  was  formulated  and  delivered  influenced  ratee  satisfaction,  which  in  turn  affected  use  of  the  feedback;  ratees 
had  more  favorable  reactions  when  they  viewed  the  rater  as  credible,  the  feedback  was  based  on  job  criteria,  and  the  rater 
refrained  from  criticism  and  engaged  in  goal  setting  with  the  ratee.  These  findings  have  implications  for  design  and  delivery 
of  performance  feedback. 
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Bettenhausen,  and  Davis  (1999)  found  that  nurses  were  more  positive  toward  peer  reviews 
when  they  perceived  the  reviews  to  be  developmental  rather  than  evaluative,  and  Farh,  Can- 
nella,  and  Bedeian  (1991)  found  that,  although  participants  had  similar  opinions  about  the 
value  and  accuracy  of  ratings,  they  were  more  likely  to  recommend  peer  feedback  for  develop¬ 
mental  than  evaluative  purposes. 


Evidence  of  Performance  Improvement  Using  360s 

360s  are  popular  in  large  part  because  participants  (raters,  ratees,  and  managers)  generally 
like  them.  But  liking  360s  is  not  the  ideal  measure  of  success  when  they  cost  an  organiza¬ 
tion  significant  time  and  resources.  As  shown  in  Table  2.1,  success  of  360s  can  be  defined  in  a 
number  of  ways.  However,  from  the  organization’s  perspective,  behavior  change  should  be  the 
ultimate  goal.  As  such,  Bracken,  Timmreck,  Fleenor,  and  Summers  (2001)  believe  that  360s 
are  successful  when  they  create  “focused,  sustained  behavior  change  and/or  skill  development 
in  a  sufficient  number  of  people  so  as  to  result  in  increased  organization  effectiveness”  (p.  4). 

Acceptance  of  the  process  by  the  raters  and  ratees  can  be  an  important  precursor  to 
behavior  change  in  that  it  can  impact  the  usefulness,  accuracy,  and  acceptance  of  the  feedback 
itself.  As  a  result,  the  other  measures  of  success  should  not  be  disregarded.  Nevertheless,  failing 
to  explore  whether  meaningful  and  desirable  behavior  change  is  occurring  could  lead  an  orga¬ 
nization  to  waste  significant  resources  on  something  with  little  improvement  in  organizational 
effectiveness. 

Research  shows  that  360  feedback  results  in  small  but  positive  improvements  in  individ¬ 
ual  performance.  A  meta-analysis  of  24  studies  found  that,  on  average,  managers’  performance 
improved  following  360  feedback;  however,  the  magnitude  of  the  improvement  was  small 
(Smither,  London,  and  Reilly,  2005).  Findings  were  similar  regardless  of  the  role  of  the  evalu¬ 
ator  (i.e.,  subordinate,  peer,  supervisor,  or  self),  whether  the  person  being  evaluated  received 
feedback  only  from  subordinates  or  from  subordinates  and  others  (i.e.,  peers  and/or  supervi¬ 
sors),  and  the  type  of  study  design  used.  “The  most  likely  explanation  for  the  small  effect  sizes 
is  that  some  feedback  recipients  are  more  likely  to  change  than  others”  (Smither,  London,  and 
Reilly,  2005,  p.  45).  This  is  consistent  with  the  results  of  a  larger  meta-analysis  of  feedback 
interventions  more  generally  (i.e.,  not  360  feedback  exclusively),  which  found  that,  while  most 
individuals  improved,  about  one-third  showed  decreases  in  performance  (Kluger  and  DeNisi, 
1996).  In  light  of  their  review  findings,  Smither,  London,  Vasilopoulos,  et  al.  (1995)  stated  the 
following: 

[I]t  is  unrealistic  for  practitioners  to  expect  large  across-the-board  performance  improve¬ 
ment  after  people  receive  multisource  feedback.  Instead,  it  appears  that  some  feedback 
recipients  will  be  more  likely  to  improve  than  others.  We  therefore  think  it  is  time  for 
researchers  and  practitioners  to  ask  “Under  what  conditions  and  for  whom  is  multisource 
feedback  likely  to  be  beneficial?”  (rather  than  asking  “Does  multisource  feedback  work?”). 

(p.  60) 

A  few  studies  have  also  shown  inconclusive  results  with  respect  to  predicting  overall  orga¬ 
nizational  performance.  For  example,  Pfau  and  Cohen  (2003)  found  that,  while  higher-quality 
human  resource  management  practices  were  positively  associated  with  firm  financial  perfor- 
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Table  2.1 

Examples  of  How  360  Success  Could  Be  Defined  from  Various  Viewpoints 


Constituent 

Type  Definitions  of  Success  Factors  Contributing  to  Success 


Rater 


Ratee 


Supervisor 


Organization 


Perception  that  feedback  was  accepted 
and  ratee  behavior  improved 
Improved  working  relationship 
No  negative  repercussions 
Improved  feedback  climate 

Received  feedback  that  was  fair,  con¬ 
structive,  valid,  actionable,  credible,  and 
reliable 

Improved  communications 
Improved  working  relationship 
Improved  feedback  climate 
Improved  performance 


Access  to  timely,  quality  feedback  appro¬ 
priate  for  use  in  performance  management 
and/or  development 

Feedback  aligned  with  organization  goals 
and/or  values  (i.e.,  relevant,  important) 
Improvement  in  ratee  behavior  consistent 
with  the  process'  objectives 
Improvement  in  ratee  workgroup 
Improved  feedback  climate 
Not  overly  intrusive  (time,  cost) 


Sustained  focused  behavioral  change  in 
large  number  of  individuals  that  leads  to 
organizational  effectiveness 
Viability  of  the  process  (sustainable) 
Improved  feedback  climate 
Legal  defensibility 
Supports  learning  environment 


Anonymity 

Ratee  accountability 

Multiple  administrations 

Policies  and  practices  communicated 

Hotline  to  report  problems 

Consistency 
Confidentiality 
Rater  training 

Rater  accountability  (i.e.,  honesty) 
Consistent  rater  selection 
Sufficient  number  of  raters  with  knowl¬ 
edge  (i.e.,  census) 

Valid,  reliable  instrument 
Organization  commitment  (e.g.,  resources) 
Access  to  raters  for  follow-up 
Development  and  action  planning 

Supervisor  receives  data  that  are  clear, 
actionable,  relevant 

Supervisor  receives  training  on  proper  use 
of  data 

Ratee  accountability  for  follow-through 
Organization  commitment  (e.g.,  resources) 
Process  is  a  business  priority 
Process  is  administered  regularly, 
consistently 

All  supervisors  held  accountable 
Valid,  reliable  instrument 
Efficient  data  collection,  reporting 

Top  management  commitment 
System-wide  implementation 
Resources  to  develop  valid  process 
Sufficient  resources  to  sustain  the  process 
Valid,  reliable  instrument 
Alignment 

Efficient,  cost-effective  data  collection, 
processing,  and  reporting 


SOURCE:  Bracken,  Timmreck,  Fleenor,  and  Summers,  2001. 


mance,  360-feedback  programs  were  associated  with  a  10.6-percent  decrease  in  shareholder 
value.  The  authors  attributed  this  finding  to  ineffective  implementation  of  360-feedback  pro¬ 
grams.  However,  the  results  were  correlational;  it  is  possible  that  companies  that  suffer  from 
a  decrease  in  shareholder  value  were  more  likely  to  implement  360-feedback  programs.  In 
addition,  shareholder  value  is  only  one  measure  of  organizational  performance,  and  it  is  not  a 
measure  that  is  particularly  relevant  to  military  organizations. 

As  noted  by  many  researchers  (see,  for  example,  Bracken  and  Rose,  2011),  however,  few 
360  efforts  ever  include  measures  of  success. 


Designing  and  Implementing  a  360-Feedback  System 

Once  an  organization  decides  that  360  feedback  is  appropriate,  numerous  design  and  imple¬ 
mentation  factors  need  to  be  considered.  Some  of  these  factors  include  the  following: 
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•  Is  the  organization  ready  for  a  360-feedback  program?  Is  360  feedback  consistent  with  the 
organizational  culture  and  goals? 

•  Will  360  feedback  be  used  for  development  only,  for  administrative  decisions  only,  or 
both? 

•  Who  should  be  involved  in  the  system  design  and  implementation? 

•  What  content  should  be  included  on  a  360-rating  form? 

•  What  types  of  ratings  should  be  used? 

•  Should  the  form  include  open-ended  questions? 

•  Should  participation  for  both  ratees  and  raters  be  voluntary  or  compulsory? 

•  Who  chooses  the  raters? 

•  How  many  raters  of  each  type  are  needed  (e.g.,  supervisors,  peers,  subordinates,  custom¬ 
ers)?  How  many  raters  should  be  invited  from  each  source? 

•  How  often  will  feedback  be  collected? 

•  How  will  results  be  presented  to  ratees  (e.g.,  aggregated  within  source  and/or  across 
sources)?  Presented  along  with  normative  data?  Presented  over  time? 

•  Who  sees  the  results? 

•  Should  raters  be  anonymous  or  identified? 

•  How  will  participants  be  trained? 

Unfortunately,  there  are  no  universal  best  answers  to  each  of  these  questions.  Instead, 
each  decision  needs  to  be  tailored  to  the  organization’s  unique  context  and  needs.  As  such, 
there  can  be  quite  a  bit  of  variability  in  the  implementation  of  360s  across  contexts  and  orga¬ 
nizations.  Table  2.2,  which  summarizes  the  results  of  the  3D  Group’s  (2013)  survey  of  360- 
feedback  practices  in  211  U.S.  and  Canadian  companies,  illustrates  some  of  that  variability. 


Table  2.2 

Summary  of  Company  Experiences  Using  360  Feedback 


Which  types  of  360  programs  does  your  organization 
use? 

Who  developed  the  survey  and  report  format? 

Individual:  The  360-degree  feedback  process  is 
made  available  on  an  as-needed  basis. 

78% 

Committee/team  (internal) 

33% 

Training  and  development:  The  360  program  is 
part  of  a  larger  training  and/or  development 
program. 

45% 

Consultant 

22% 

Enterprise-wide:  All  or  nearly  all  of  the 
managers  within  the  organization  participate. 

23% 

Internal  committee  or  team  with  a  consultant 

20% 

Department-wide:  Specific  departments  or 
units  go  through  the  360  process. 

17% 

Bought  "off  the  shelf" 

20% 

Other 

5% 

How  many  points  are  on  your  rating  scale? 

Does  your  rating  scale  focus  on: 

5 

87% 

Agreement  (such  as  "strongly  agree"  to 
"strongly  disagree") 

49% 

4 

6% 

Effectiveness  (such  as  "not  effective"  to  "very 
effective") 

31% 

6 

2% 

Frequency  (such  as  "always"  or  "daily") 

23% 

7 

2% 

Importance  (such  as  "very  important"  for  the 
job) 

17% 

Other  (3,  9,  or  10) 

3% 

Other 

7% 
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Table  2.2 — Continued 


How  many  questions  does  your  survey  contain? 

How  many  open-ended  questions  are  included? 

11  to  40 

53% 

Two 

22% 

41  to  69 

21% 

Five 

20% 

Up  to  10 

17% 

Three 

16% 

70  to  100 

6% 

Zero 

15% 

Over  100 

3% 

Six  to  ten 

11% 

One 

7% 

Four 

7% 

Over  ten 

3% 

How,  if  at  all,  do  you  present  variation  in  scores  across 
raters? 

What  standard  measure,  if  any,  do  you  have  to  which 
feedback  recipients  can  compare  their  results? 

Illustrate  graphically 

37% 

Organization  norms 

34% 

Raw  scores — frequency  distribution 

33% 

No  standard  measure 

30% 

None 

14% 

Previous  years'  scores 

26% 

Standard  deviation 

13% 

National  norms 

19% 

Who  chooses  raters? 

How  do  you  train  your  raters  to  complete  surveys? 

Participant  with  manager  approval 

42% 

Test  included  in  survey 

56% 

Human  resources  (based  on  organizational 
chart) 

25% 

Web  training 

19% 

Participant  only 

17% 

In-person  training 

13% 

Manager  only 

10% 

Email 

6% 

Teleconference 

6% 

Who  receives  a  copy  of  the  report? 

What,  if  any,  is  the  minimum  number  of  raters  per  rater 
category  (e.g.,  peers,  direct  reports)  for  data  to  show 
in  a  report? 

Participant 

96% 

Three 

38% 

Human  resources 

60% 

No  minimum 

19% 

Participant's  direct  manager 

59% 

Two 

11% 

360  coach 

24% 

Four 

10% 

Five 

10% 

Six 

7% 

More  than  six 

5% 

SOURCE:  3D  Group,  2013. 
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Fortunately,  however,  there  is  an  enormous  amount  of  theoretical  and  empirical  research 
addressing  the  details  associated  with  the  design  and  implementation  of  360s  that  can  help 
guide  organizations  in  making  many  of  these  decisions.  Because  covering  all  of  the  relevant 
issues  is  far  beyond  the  scope  of  this  report,  we  direct  organizations  to  consult  the  existing  lit¬ 
erature  about  best  practices  in  360  feedback  to  guide  their  efforts.  Sources  do  vary,  however, 
in  the  extent  to  which  they  provide  sound,  research-based  information  and  recommendations. 
The  Handbook  of  Multisource  Feedback  (Bracken,  Timmreck,  and  Church,  2001)  is  a  particu¬ 
larly  comprehensive  and  useful  resource  to  support  implementation;  it  consists  of  chapters 
addressing  a  wide  range  of  topics,  written  by  leading  experts  in  360-feedback  research  and 
practice.  We  direct  interested  readers  to  consult  that  resource  as  a  starting  point. 


CHAPTER  THREE 


Performance  Evaluation  and  Promotion  Processes  in  the  Military 
Services 


Performance  evaluation  in  the  military  has  traditionally  consisted  of  top-down  evaluations  in 
which  supervisors  rate  their  subordinates.  With  this  top-down  approach,  there  is  no  opportu¬ 
nity  for  subordinates  to  contribute  content  directly  and  systematically  to  performance  evalu¬ 
ation.  The  current  performance  evaluation  system  also  does  not  provide  a  formal  opportunity 
for  review  of  a  leader’s  performance  by  peers  of  similar  rank.  In  an  organization  as  hierarchical 
as  the  military,  this  could  mean  that  important  facets  of  leadership  and  managerial  perfor¬ 
mance  are  omitted  from  performance  evaluation  information.  A  360-degree  assessment  is  one 
method  that  some  think  would  be  useful  to  help  fill  in  those  gaps.  It  is  a  tool  that  the  mili¬ 
tary  services  use  to  varying  degrees,  which  we  discuss  in  the  following  chapter.  As  context  for 
considering  how  or  whether  360s  might  be  used  more  broadly  as  a  component  of  performance 
evaluation  and  promotion  processes  in  the  military  services,  we  begin  in  this  chapter  with  a 
description  of  those  systems.1 


Performance  Evaluation  Systems 

Officer  performance  evaluation  reports  are  governed  primarily  by  regulations  issued  by  the 
individual  services.  Each  service  has  developed  its  own  philosophy,  policies,  and  procedures  for 
evaluating  the  performance  and  potential  of  its  officers.  There  are,  however,  three  provisions  of 
law  applicable  to  all  services  that  prescribe  specific  action  regarding  performance  evaluations. 
The  first  requirement — found  in  Title  10  of  the  United  States  Code;  Part  II,  Defense  Acquisi¬ 
tion  Positions',  Chapter  87,  Defense  Acquisition  Workforce',  Section  1722,  Career  Development — 
addresses  performance  appraisals  reviewed  by  a  person  serving  in  the  same  acquisition  career 
field.2  The  other  two  provisions  impose  a  requirement  for  the  chairman  of  the  Joint  Chiefs  of 


1  This  overview  of  current  performance  evaluation  and  promotion  processes  in  the  military  services  is  based  primarily  on 
a  document  review  of  laws,  policies,  and  procedures  that  was  verified  and  expanded  during  interviews  with  representatives 
from  each  of  the  services. 

2  This  section  requires  the  secretary  of  each  military  department  to  “provide  an  opportunity  for  review  and  inclusion  of 
any  comments  on  any  appraisal  of  the  performance  of  a  person  serving  in  an  acquisition  position  by  a  person  serving  in  an 
acquisition  position  in  the  same  acquisition  career  field.”  This  section  is  implemented  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (Department  of  Defense  [DoD]  Instruction  5000.66,  2005). 
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Staff  to  assess  the  performance  of  an  officer  as  a  member  of  the  Joint  Staff  or  other  joint  duty 
assignments.3 

While  each  service  prescribes  its  own  procedures  and  criteria  for  evaluating  the  perfor¬ 
mance  and  potential  of  its  officers,  there  is  some  commonality  among  the  services: 

•  Evakiation  reports.  Three  services  prepare  reports  for  O-ls  through  0-8s;  the  Army  pre¬ 
pares  reports  only  for  O-ls  through  0-7s. 

•  Annual  reporting  cycle.  Most  reports  are  prepared  annually,  but  the  services  have  proce¬ 
dures  to  accommodate  unique  situations  that  may  affect  this  annual  cycle. 

•  Top-down  evaluations.  The  person  who  prepares  the  assessment  and  subsequent  reviews  is 
in  the  supervisory  chain  above  the  officer  being  evaluated.  Peers  and  subordinates  have  no 
input  into  the  preparation  of  performance  evaluations. 

•  Rating  chain.  Three  services  use  multiple  raters;  the  Navy  uses  a  single  rating  official. 

•  Service  advisors.  Advisors  are  designated  to  assist  rating  officials  when  the  officer  being 
evaluated  is  assigned  to  an  organization  or  command  outside  the  officer’s  service  and  the 
rating  chain  does  not  include  an  officer  in  the  same  service  as  the  officer  being  evaluated. 

•  Evaluation  criteria.  Each  service  has  established  its  own  set  of  competencies  and  attributes 
by  which  it  measures  an  officer’s  performance  and  potential  to  assume  greater  responsibil¬ 
ity.  Leadership  is  the  only  evaluation  criterion  common  to  all  services. 

•  Written  comments.  Rating  officials  provide  written  comments  on  the  performance  of  the 
officer  being  evaluated. 

•  Assessment  metric.  A  rating  scale  is  used  for  officers  in  certain  grades,  with  those  grades 
determined  by  service  policy. 

•  Promotion  recommendation.  Three  services  include  an  assessment  of  an  officer’s  potential 
to  assume  greater  responsibility;  the  Air  Force  accomplishes  this  function  on  a  separate 
form. 

•  Signature.  The  officer  being  evaluated  must  sign  the  performance  report  after  all  reviews 
are  complete. 

•  Appeal  process.  Each  service  has  a  multistep  process  that  allows  an  officer  to  raise  concerns 
about  an  evaluation.  The  final  avenue  for  redress  is  an  application  to  the  service  board  for 
correction  of  military  or  naval  records. 

•  Counseling.  Each  service  includes  individual  counseling  as  part  of  the  performance  evalu¬ 
ation  cycle.  Three  services  include  counseling  as  an  integral  part  of  the  evaluation  process; 
the  Marine  Corps  views  counseling  as  distinct  and  separate,  but  complementary  to  the 
performance  evaluation  system. 

Table  3.1  provides  an  overview  of  the  similarities  and  differences  of  the  major  elements  of 
performance  evaluation  systems  used  by  each  of  the  military  services. 


3  The  first  of  these  two  provisions  calls  for  the  chairman  of  the  Joint  Chiefs  to  provide  the  assessment  when  an  officer 
has  been  nominated  to  a  position  of  importance  and  responsibility,  designated  by  the  president,  which  carries  the  grade  of 
general,  admiral,  lieutenant  general,  or  vice  admiral  (Section  601  of  Title  10,  United  States  Code).  The  provision  calls  for 
an  assessment  when  an  officer  has  been  nominated  as  a  commander  of  a  combatant  command;  commander,  U.S.  Forces, 
Korea;  or  deputy  commander,  U.S.  European  Command,  but  only  if  the  commander  of  that  command  is  also  the  Supreme 
Allied  Commander,  Europe  (Section  604  of  Title  10,  United  States  Code).  Under  the  second  provision,  the  chairman  may 
also  include  consideration  of  other  aspects  of  the  officer’s  performance  as  the  chairman  considers  appropriate. 
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Table  3.1 

Comparison  of  the  Military  Services'  Performance  Evaluation  Systems 


Service 

Evaluation  Form 

Rating  Chain 

Evaluates 

Performance 

Evaluates 

Potential 

Evaluation 

Approach 

Appeal 

Process 

Counseling3 

Army 

•  Company  grade 

•  Field  grade 

•  Strategic  gradeb 

•  Academic 

•  Rater 

•  Intermediate 
rater 

•  Senior  rater 

Yes 

Yes 

Rating  scale 
(company  and 
field  grades)b 
Narrative 

Yes 

Yes 

Navy 

•  Ensign  to  captain 

•  Rear  admiral 

•  Rating  official 

Yes 

Yes 

Rating  scale 
Narrative 

Yes 

Yes 

Marine 

Corps 

•  Lieutenant  to 
major  general0 

•  Reporting  senior 

•  Reviewing  officer 

•  Third  officer 
sighter 

Yes 

Yes 

Rating  scale 
(0-5  and 
below) 
Narrative01 

Yes 

Separately6 

Air 

Force 

•  Lieutenant  to 
colonel 

•  Brigadier  to 
major  general 

•  Academic 

•  Rater 

•  Additional  rater 

•  Reviewer 

Yes 

In  the 
narrative^ 

"Does 
not  meet 
standards"9 
(0-6  and 
below) 
Narrative 

Yes 

Yes 

SOURCES:  Army  Regulation  623-3,  2014;  Chief  of  Naval  Personnel,  2011;  Commandant  of  the  Marine  Corps,  2010; 
Air  Force  Instruction  36-2406,  2005;  and  representatives  who  provided  input  in  our  interviews. 
a  Frequency  and  method  of  counseling  vary  among  the  services.  The  Army  and  Air  Force  use  separate  forms  to 
facilitate  counseling  and  require  counseling  to  be  conducted  face  to  face  if  at  all  possible.  The  Navy  leaves  the 
counseling  method  to  the  commanding  officer  or  officer  in  charge. 

b  In  the  Army,  Air  Force,  and  Marine  Corps,  officers  in  the  pay  grades  of  0-1  to  0-3  are  referred  to  as  company 
grade,  0-4  to  0-6  are  field  grade,  and  0-7  and  higher  are  strategic  grade  (general  officers).  The  equivalent 
officer  groupings  in  the  Navy  are  called  junior  grade,  midgrade,  and  flag. 

c  Marine  Corps  generals  and  colonels:  Only  specified  information  in  the  administrative  section  is  completed; 
evaluation  is  submitted  using  letter  format. 

d  Marine  Corps  colonels:  Consideration  and  evaluation  of  attributes  are  addressed  in  the  narrative  letter. 
e  Marine  Corps:  Counseling  is  a  separate  but  complementary  process  to  the  performance  evaluation  system. 
f  Air  Force  0-6  and  below:  Uses  a  separate  process  specifically  to  make  a  promotion  recommendation. 

9  The  Air  Force  system  provides  for  an  evaluation  in  several  criteria  whereby  an  officer  is  rated  as  "meets"  or 
"does  not  meet"  standards  in  each  area. 


Tables  3.2  and  3.3  compare  the  evaluation  criteria  established  by  each  of  the  military 
services  to  evaluate  its  officers.  To  help  illustrate  some  of  the  similarities  and  differences,  we 
sorted  each  service’s  evaluation  criteria  into  three  conceptual  categories  (professional  or  tech¬ 
nical  expertise,  personal  qualities,  and  leadership  and  command  climate).4  The  words  in  bold 
type  are  the  criteria,  and  the  description  following  the  titles  provides  more  detail  on  what  the 
rater  actually  evaluates.  Because  some  criteria  are  relatively  self-explanatory,  a  description  is 
not  provided  for  all  criteria,  but  each  service  has  a  more  detailed  description  of  the  attributes 
for  each  criterion  in  its  governing  regulation.  Table  3.2  shows  the  criteria  for  0-6s  and  below; 
Table  3.3  for  general  and  flag  officers. 

While  each  service  has  its  own  approach  to  and  criteria  for  assessing  performance,  they  all 
achieve  similar  results.  The  system  establishes  the  performance  characteristics  that  are  impor¬ 
tant  to  the  service  and  documents  how  well  the  officer  met  those  standards.  That  information 


4 


Our  category  groupings  are  not  necessarily  reflective  of  the  services’  groupings  of  these  criteria. 
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Table  3.2 

0-6  and  Below  Performance  Evaluation  Criteria 


Service 

Professional/ 

Technical  Expertise 

Personal  Qualities 

Leadership/ 

Command  Climate 

Army 

•  Achieves:  gets  results,  pri- 

•  Character:  adheres  to  Army 

•  Leads:  leads  others. 

oritizes  limited  resources 

values,  empathy,  warrior 

leads  by  example,  builds 

to  accomplish  the  mis- 

ethos/service  ethos,  disci- 

trust,  extends  influence 

sion,  accomplishes  mission 

pline;  supports  SHARP,  EEO/ 

beyond  chain  of  command, 

consistently 

EO 

communicates 

•  Presence:  military  bearing, 

•  Develops:  creates  a  posi- 

fitness,  confidence,  resil- 

tive  command/workplace 

ience,  professional  bearing 

environment,  prepares  self, 

•  Intellect:  mental  agility, 

develops  others,  stewards 

sound  judgment,  innovation, 

the  profession 

interpersonal  tact,  expertise 

Navy 

•  Professional  expertise: 

•  Military  bearing/character: 

•  Teamwork 

knowledge,  proficiency, 

appearance,  conduct,  physi- 

•  Leadership:  organizes,  moti- 

qualifications 

cal  fitness,  adherence  to 

vates,  develops  others  to 

•  Mission  accomplishment  and 

Navy  core  values 

achieve  goals 

initiative 

•  Command  climate 

•  Tactical  performance 

•  Equal  opportunity 

(warfare-qualified  officers 

only) 

Marine 

•  Mission  accomplishment: 

•  Individual  character:  effec- 

•  Leadership:  leading/ 

Corps 

performance,  proficiency 

tiveness  under  stress,  cour- 

developing  subordinates. 

•  Fulfillment  of  evaluation 

age,  initiative 

ensuring  well-being  of  sub- 

responsibilities 

•  Intellect  and  wisdom:  deci- 

ordinates,  setting  the  exam- 

sionmaking  ability,  judgment 

pie,  communication  skills 

Air 

•  Job  knowledge 

•  Judgment  and  decisions 

•  Leadership  skills:  fair  and 

Force 

•  Organizational  skills:  plans, 

•  Physical  fitness 

consistent,  fosters  team- 

coordinates,  schedules,  and 

•  Professional  qualities:  loy- 

work,  sets/enforces  stan- 

uses  resources  effectively 

alty,  honesty,  discipline,  ded- 

dards,  motivates  subordi- 

ication,  integrity,  officership 

nates,  promotes  healthy 

organizational  climate 

•  Communication  skills 

SOURCES:  Army  Regulation  623-3,  2014;  Chief  of  Naval  Personnel,  2011;  Commandant  of  the  Marine  Corps,  2010; 
Air  Force  Instruction  36-2406,  2005;  and  representatives  who  provided  input  to  our  interviews. 

NOTES:  SHARP  =  Sexual  Harassment/Assault  Response  and  Prevention  Program;  EEO  =  equal  employment 
opportunity;  EO  =  equal  opportunity. 


is  used  in  turn  for  a  variety  of  personnel  actions,  most  notably  during  selection  board  delibera¬ 
tions  in  which  promotions  decisions  are  made. 


Promotion  Process 

Unlike  performance  evaluation  systems,  the  process  for  selecting  and  promoting  officers  is 
prescribed  in  law.5  DoD  issues  implementation-related  policies,  and  each  service  publishes 
regulations  specifying  how  to  comply  with  the  law  and  implement  DoD  policies.  The  laws 
governing  officer  promotion  in  the  military  services  fall  primarily  under  Chapters  35  and  36  of 
Title  10,  United  States  Code.  Chapter  36  contains  most  of  the  promotion  process  provisions; 
Subchapter  I  prescribes  the  requirements  for  selection  boards  (when  they  are  to  be  convened, 


5  The  promotion  laws  described  in  this  section  are  relevant  to  active-duty  officers.  Additional  chapters  of  Title  10  of  the 
United  States  Code  cover  promotions  for  reserve  officers. 
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Table  3.3 

General/Flag  Officer  Performance  Evaluation  Criteria 


Service 

Professional/ 

Technical  Expertise 

Personal  Qualities 

Leadership/ 

Command  Climate 

Army 

•  Achieves:  accomplishes  the 

•  Character:  adheres  to  Army 

•  Leads:  leads  by  example, 

mission,  maintains/builds 

values,  warrior  ethos,  moral 

builds  trust,  extends  influ- 

organization's  capabilities 

and  ethical  qualities 

ence  beyond  chain  of 

•  Presence:  projects  a  com- 

command 

manding  presence 

•  Develops:  creates  a  positive 

•  Intellect:  mental  agility, 

command  climate,  seeks  self- 

sound  judgment,  innovation 

improvement,  steward  of 

the  profession 

Navy 

•  Fiscal  planning/ 

•  Vision/strategic  thinking 

•  Communication  skills 

organizational  skills 

•  Geopolitical  fluency 

•  Leadership/judgment 

•  Professional  growth 

•  Military  bearing 

•  Leading  change 

•  Mission  accomplishment 

•  Personal  growth 

•  Operational/professional 

competence 

Marine 

•  None  specified 

Corps 

Air 

•  None  specified 

Force 

•  Comment  on  professional  and  personal  characteristics 

SOURCES:  Army  Regulation  623-3,  2014;  Chief  of  Naval  Personnel,  2011;  Commandant  of  the  Marine  Corps,  2010; 
Air  Force  Instruction  36-2406,  2005;  and  representatives  who  provided  input  to  our  interviews. 


composition  of  the  board,  information  to  be  furnished  to  the  selection  board,  reporting  and 
approval  requirements),  and  Subchapter  II  deals  with  promotions  (criteria  a  selection  board 
must  consider  and  details  of  the  promotion  process,  such  as  eligibility  for  promotion).  Two  sec¬ 
tions  of  Chapter  35  prescribe  the  procedures  for  appointment  of  officers  to  certain  three-  and 
four-star  positions.  Section  662  of  Chapter  38  prescribes  minimum  promotion  objectives  for 
officers  serving  or  who  have  served  on  the  Joint  Staff  or  have  been  designated  as  joint  qualified. 
In  addition,  sections  in  other  chapters  of  Title  10  contain  specific  requirements  with  which 
promotion  selection  boards  must  comply  when  considering  officers  for  promotion. 

OSD  and  Service  Guidance 

The  Secretary  of  Defense  has  issued  regulations  to  implement  the  provisions  of  law  regarding 
selection  boards  and  the  promotion  process.  The  two  primary  instructions  are  DoD  Instruc¬ 
tion  1320.14,  Commissioned  Officer  Promotion  Program  Procedures  (2013)  and  DoD  Instruc¬ 
tion  1320.13,  Commissioned  Officer  Promotion  Reports  (2014).  Each  of  the  services  has  issued 
implementing  instructions  regarding  the  promotion  process  as  prescribed  in  Title  10  and  DoD 
policy.  Of  particular  note,  each  service  policy  specifies  the  information  provided  to  selection 
boards  as  prescribed  in  Section  615  of  Title  10: 

•  Army  Regulation  600-8-29,  Officer  Promotions  (2005),  states  that  promotion  boards  are 
to  receive  the  “performance  portion  of  the  OMPF  [Official  Military  Personnel  File].” 
Army  Regulation  623-3,  Evaluation  Reporting  System  (2014),  specifies  that  evaluations 
will  be  maintained  in  the  soldier’s  Official  Military  Personnel  File. 

•  Secretary  of  the  Navy  Instruction  1420. IB,  Promotion,  Special  Selection,  Selective  Early 
Retirement,  and  Selective  Early  Removal  Boards  for  Commissioned  Officers  of  the  Navy  and 
Marine  Corps  (Secretary  of  the  Navy,  2006),  states  that  the  Chief  of  Naval  Personnel  and 
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the  commandant  of  the  Marine  Corps  “shall  supply  all  pertinent  records  of  each  officer 
to  be  considered  by  the  board.  Such  records  shall  include  all  documents,  including  fitness 
reports.  .  .  .” 

•  Air  Force  Instruction  36-2501,  Officer  Promotion  and  Selective  Continuation  (2014),  speci¬ 
fies  that  the  name  and  officer  selection  record  is  information  given  to  the  board.  Air  Force 
Instruction  36-2406,  Officer  and  Enlisted  Evaluation  Systems  (2005),  states  that  an  evalua¬ 
tion  becomes  a  matter  of  record  once  it  has  been  filed  in  the  officer  selection  record. 

These  requirements  bear  directly  on  decisions  concerning  the  use  of  360-degree  feedback 
in  the  military  services.  Specifically,  if  360s  become  a  formal  part  of  the  evaluation  system 
and,  thus,  are  included  in  officer  performance  evaluation  reports,  the  results  of  the  360s  would, 
under  current  law,  be  included  in  the  information  provided  to  selection  boards.  Whether  this 
is  a  desirable  practice  should  be  considered  before  making  a  decision  to  use  360s  for  evaluation 
purposes  in  the  military,  as  is  discussed  later  in  this  report. 


Summary 

This  review  demonstrates  that  each  service  sets  its  own  performance  evaluation  policies,  fol¬ 
lowing  standards  that  best  meet  its  respective  needs.  That  the  services  tailor  their  performance 
evaluation  systems  is  an  important  attribute  to  keep  in  mind  when  considering  whether  and 
how  to  further  implement  360s  in  DoD.  For  example,  should  the  department  decide  to  man¬ 
date  the  use  of  360s,  even  in  a  limited  way,  such  as  for  select  leadership  positions,  it  may  be 
advisable  to  allow  the  services  to  determine  an  implementation  approach  that  aligns  with  their 
current  evaluation  processes. 

In  contrast,  the  law  controls  the  promotion  process.  Numerous  provisions  establish  the 
requirements  for  determining  eligibility,  establishing  and  conducting  a  selection  board,  and 
specifying  the  process  that  must  be  followed  for  an  officer  to  be  promoted.  Perhaps  the  most 
important  attribute  of  the  promotion  process  as  pertains  to  the  use  of  360s  is  the  fact  that  per¬ 
formance  evaluations  are  a  formal  part  of  an  officer’s  selection  record  and  thereby  provided  to 
selection  boards  for  consideration.  Nothing  in  law  precludes  the  use  of  a  360  or  the  inclusion  of 
360s  in  promotion  considerations.  However,  should  360s  be  mandated  as  part  of  performance 
evaluations,  they  would  necessarily  be  provided  to  selection  boards  unless  current  laws  are 
modified.  This  too  is  an  important  consideration  in  determining  the  best  use  of  360s  within 
the  department. 

The  chapters  that  follow  review  current  use  of  360s  in  the  military  services,  discuss  vari¬ 
ous  uses  of  360s,  and  evaluate  these  alternatives  within  the  military  context. 


CHAPTER  FOUR 


360-Feedback  Systems  Currently  in  Use  in  the  Military 


All  of  the  services  have  channels  for  feedback  to  personnel  outside  of  the  traditional  top-down 
review  systems.  These  can  be  formal,  such  as  the  Marine  Corps’  “Requesting  Mast”  (a  Marine 
can  speak  privately  about  his  or  her  concerns  with  someone  several  ranks  higher);  informal, 
such  as  casual  conversation  and  discussions  across  ranks  and  peers;  event-specific,  such  as  after¬ 
action  reports  in  the  Army  and  the  Marine  Corps  (where  debriefs  are  conducted  after  a  mis¬ 
sion);  or  environment  focused,  such  as  command  climate  surveys,  which  focus  on  the  qualita¬ 
tive  features  of  command,  such  as  integration  and  honesty.  360s  are  currently  just  one  more 
structure  for  aggregating  feedback  from  outside  the  direct  command  chain. 

Each  service  and  the  Joint  Staff  are  using  some  version  of  a  360.  Table  4.1  summarizes 
several  features  of  the  360  programs  we  identified  in  our  military  stakeholder  interviews.1  As 
shown  in  the  table,  there  are  large  areas  of  commonality  in  the  utilization  of  360s  across  the 
services.  For  example,  each  service  is  using  360s  in  some  form  for  development,  and  none  has 
360s  in  place  for  performance  evaluation.  However,  there  are  some  clear  differences  as  well. 
For  example,  the  Army,  with  the  most  experience  by  far,  has  spent  14  years  developing  its 
MSAF  program,  which  now  has  widespread  implementation.  The  Army  is  also  the  only  service 
that  requires  that  each  officer  complete  a  360. 

Other  services,  such  as  the  Marine  Corps,  which  is  just  starting  a  pilot,  have  far  less  expe¬ 
rience,  and  their  programs  tend  to  be  smaller  and  narrowly  targeted.  Most  are  directed  either 
at  senior  leadership  or  toward  high-potential  officers  as  part  of  the  military  education  system. 
A  number  of  the  programs  are  in  early  stages  of  development  and  are  being  piloted  on  small 
subsets  of  the  services’  respective  populations.  All  of  the  programs  are  developmental  in  nature, 
and  their  small  scale  typically  allows  for  personalized  coaching  support  in  the  interpretation 
and  utilization  of  the  results.  Despite  the  shared  developmental  focus,  there  is  variation  across 
these  programs  in  how  widely  the  results  of  the  assessments  are  shared.  In  many  cases,  only  the 
individual  and  his  or  her  coach — if  he  or  she  elects  to  have  one — see  the  results.  However,  in 
both  the  Air  Force  and  the  Joint  Staff,  the  chief  and  chairman  (respectively)  can  access  general 
officers’  360  results. 

Given  that  the  Army  has  the  largest  existing  360  effort  and  the  NDAA  specifically  points 
to  it  as  an  exemplar,  the  Army’s  programs  are  described  in  detail  below.  Following  that,  we 
describe  some  other  existing  efforts  in  the  services  that  could  serve  similar  purposes  to  a  360. 


1  Although  this  is  a  fairly  comprehensive  list  of  the  formalized  360  efforts  in  the  services,  it  is  likely  that  there  are  other 
efforts  scattered  throughout  the  services  (such  as  those  embedded  in  larger  training  programs  or  that  are  taking  place  in 
other  informal  development  settings)  that  are  not  captured  in  the  table. 
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Table  4.1 

Examples  of  Existing  360  Efforts  in  Each  Service 


Service 

Tools  Used 

For  Whom? 

Who  Provides 
Ratings? 

Is  It  Required,  and 
How  Often? 

Who  Sees  the 
Results? 

Air 

Force 

Air  Force 
Leadership 

Mirror 

Many  Air  Force 
populations 

Participant  selects 
(subordinates, 
peers,  and 
superiors) 

On  taking  grade 

Individual 

Air  Force  General 
Officer  360 

General  officers 

Any  general  officer, 
command  chief 
master  sergeants, 
colonels,  GS-15s, 
Senior  Executive 
Service  members 

Yes,  annually 

Individual,  chief 
and  vice  chief  of 
the  Air  Force 

CCL  Executive 
Dimensions 

Lieutenant  generals  and 
Tier-3  Senior  Executive 
Service 

Participant  selects 

Taken  at  CCL  at 
career  milestone 

Individual  with 
the  option  of 
coaching 

CCL  Benchmarks 

Select  colonels,  GS-15, 

E-9 

Participant  selects 

Taken  at  CCL  at 
career  milestone 

Individual  with 
coaching 

Army 

Leader  360 
(MSAF) 

All  officers  (0-6  and 
below),  enlisted  and 
Department  of  the  Army 
civilians 

Participant  selects 
(CAC  holders) 

Yes,  once  every 
three  years  and 
tracked  with 
checkbox  on 
Officer  Evaluation 
Report 

Individual  with 
the  option  of 
coaching 

Commander  360 
(MSAF) 

Battalion  and  brigade 
commanders  on  the 
Centralized  Selection 

List 

Rater  selects, 
commander  can 
nominate  two  for 
inclusion 

Yes,  three  to 
six  months  and 
15-18  months  into 
command 

Commander 
feedback  report, 
must  be  reviewed 
with  rater  (who  is 
trained  to  provide 
coaching) 

Peer  and 

Advisory 

Assessment 

Program 

General  officers 

Anyone  in  general 
officer  corps  who 
feels  capable  of 
providing  feedback 

Yes,  annually 

Individual  with 
the  option  of 
coaching,  Army 
GOMO  reviews 
results  for  red 
flags 

General  Officer 
360 

Those  promoted  to 
two-,  three-,  and  four- 
star  posts 

Participant  selects 
with  Army  GOMO 
approval 

When  promoted 
to  next  general 
officer  post, 
approx,  every 
three  years 

Individual  with 
the  option  for 
coaching,  Army 
GOMO  reviews 
results  for  red 
flags 

Navy 

360  Assessment 
at  Navy 
Leadership  and 
Ethics  Center 

Officers  screened  for 
command 

Participant  selects 

On  command 
selection 

Individual  with 

one-on-one 

feedback  from 
certified  coach  at 
the  training 

Surface 

Community  Pilot 
360  Program 

Department  heads 

Participant  selects 

Early  career 
(about  seven 
years  of  service) 

Individual  with 
debriefing  by  a 
certified  feedback 
coach  at  Surface 
Warfare  Officer 
Department  Head 
School 

New  Flag 

Officer  and 

Senior  Executive 
Symposium 

New  flag  officers 

Participant  selects 

A  week-long 
training  course 

Debriefed  by 
executive  coaches 
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Table  4.1 — Continued 


Service 

Tools  Used 

For  Whom? 

Who  Provides 
Ratings? 

Is  It  Required,  and 
How  Often? 

Who  Sees  the 
Results? 

Naval  Post- 
Graduate  School 
Flag  Officer  360 

Rear  admiral  (upper  half) 
and  vice  admiral 

Participant  selects 

While  attending  a 
customized  course 
specifically  geared 
toward  preparing 
them  for  their 
next  assignment 

Debriefed  by 
executive  coaches 

CCL  Flag  Officer 
360 

Vice  admirals  and 
admirals 

Participant  selects 

At  the  Leadership 
at  the  Peak  course 

Debriefed  by 
executive  coaches 

Marine 

Corps 

Center  for  Army 
Leadership  MSAF 
360  (Pilot) 

Colonel/lieutenant 
colonel  commanders 

Participant  selects 
(CAC  holders) 

No,  pilot  (MSAF) 
run  once  at 
Commandant's 
Commanders 
Program 

Individual  with 
the  option  of 
coaching 

CCL  Executive 
Dimensions 

Lieutenant  generals 
and  Tier-3  Senior 
Executive  Service 

Participant  selects 

Taken  at  CCL  at 
career  milestone 
during  the 
Leadership  at  the 
Peak  course 

Individual  with 
the  option  of 
coaching 

Joint 

Joint  360 

Joint  Staff  and 
combatant  command 
general  and  flag 
officers 

Participant  selects 
(subordinates, 
peers,  and 
superiors) 

Yes,  six  to  nine 
months  after 
taking  post; 
again  in  two  years 

Chairman, 
deputy  chair,  and 
director  J7  may 
request  to  see  an 

individual 
Participant  also 
receives  report 
and  may  choose 
self-review, 
review  with 
coach,  supervisor, 
or  a  senior 
mentor 


NOTES:  Table  is  populated  based  on  interviews  conducted  by  RAND  researchers  and  supplemented  with  data 
provided  by  the  office  of  the  Senior  Advisor  to  the  Secretary  of  Defense  for  Military  Professionalism  in  OSD. 
GS  =  General  Schedule  pay  scale;  CCL  =  Center  for  Creative  Leadership;  CAC  =  Common  Access  Card;  GOMO  = 
General  Officer  Management  Office. 


The  Army's  Multi-Source  Assessment  and  Feedback  Program 

The  Army  currently  operates  the  most  expansive  360  program,  known  as  the  Multi-Source 
Assessment  and  Feedback  (MSAF)  program.  The  MSAF  is  intended  to  enhance  a  leader’s 
adaptability  and  self-awareness,  to  identify  strengths  and  weaknesses,  and  to  prepare  the  indi¬ 
vidual  for  future  leadership  responsibilities.  The  assessment  gathers  feedback  from  multiple 
sources — peers,  subordinates,  and  superiors — using  questions  that  focus  on  core  leader  com¬ 
petencies  and  supporting  leadership  behaviors  as  defined  in  Army  leadership  doctrine  (Army 
Doctrine  Reference  Publication  6-22,  2012).  The  results  are  used  to  establish  self-development 
goals  and  self-improvement  courses  of  action,  and  participating  leaders  have  the  option  of 
gaining  assistance  in  this  process  from  a  coach.  The  Army  governing  regulation  states  that  this 
program  is  separate  and  distinct  from  formal  performance  evaluations  and  personnel  manage¬ 
ment  practices.  Although  results  are  not  made  available  to  the  rating  chain,  whether  an  indi¬ 
vidual  completed  the  MSAF  and  the  date  of  the  most  recent  assessment  are  noted  in  annual 
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performance  evaluations  (i.e.,  the  Officer  Evaluation  Report)  for  lieutenants  through  colonels, 
although  the  results  are  accessible  only  by  the  individual  service  member. 

Every  officer  through  0-6,  noncommissioned  officer,  and  all  Department  of  the  Army 
civilians  must  complete  a  360  review  at  least  every  three  years.  There  are  multiple  versions  of 
the  tool  and  multiple  channels  for  its  administration.  Every  version  is  developmental  in  nature, 
reinforces  the  same  core  principles  as  outlined  by  Army  leadership  doctrine,  and  is  integrated 
into  a  central  hub — the  Virtual  Improvement  Center.  The  current  program  has  over  a  decade 
of  implementation  history  and  development  and  continues  to  evolve  today.  Within  the  past 
two  years,  the  Army  has  restructured  its  standard  instrument  and  broadened  its  technical 
scope.  This  process  is  continuing  to  expand  the  acceptance  of  360s  and  foster  a  culture  of  feed¬ 
back  for  the  Army. 

MSAF  360  Implementation 

The  MSAF  program  is  available  to  any  service  member  or  Department  of  the  Army  civil¬ 
ian  with  a  CAC.  Each  individual  is  tasked  with  logging  in  to  the  system;  completing  a  self- 
assessment;  and  inviting  peers,  subordinates,  and  superiors  to  provide  feedback.  According 
to  the  Center  for  Army  Leadership,  completing  the  online  instrument  takes  between  ten  and 
12  minutes.  Prior  to  beginning  the  process,  individuals  are  encouraged  to  take  online  train¬ 
ing  modules  that  outline  the  purpose  and  importance  of  the  MSAF,  the  process  of  utilizing 
the  MSAF  for  self-development,  and  training  for  raters  on  how  to  accurately  assess  an  officer’s 
behavior.  The  minimum  number  of  required  raters  is  three  superiors,  five  peers,  and  five  subor¬ 
dinates,  though  participants  are  informed  that  the  quality  of  the  assessment  result  is  improved 
with  increased  number  of  assessors.  Assessors  can  be  drawn  from  across  the  armed  services,  and 
the  Army  is  increasingly  streamlining  the  process  to  gather  feedback  from  multiple  branches. 

After  the  survey  is  completed,  the  individual  being  reviewed  receives  an  individual  feed¬ 
back  report  that  contains  average  ratings  from  each  rater  group — peers,  subordinates,  supe¬ 
riors,  and  others — as  well  as  the  free-form  comments.  The  individual  reports  are  maintained 
exclusively  for  the  eyes  of  the  user.  The  MSAF  portal,  which  provides  the  tool  and  houses  the 
report,  also  contains  links  to  development  materials,  such  as  videos,  doctrines,  and  further  self- 
assessments  housed  in  the  Virtual  Improvement  Center.  The  Army’s  objective  is  to  continue 
to  improve  the  development  center  so  that  feedback  results  are  immediately  linked  to  relevant 
training,  videos,  or  policy  documents. 

In  addition,  all  individuals  are  encouraged  to  seek  coaching  and  mentorship.  The  Army 
provides  access  to  full-time  coaches — professional  outsiders,  as  well  as  retired  personnel  trained 
for  the  task — and  encourages  the  rated  individual  to  discuss  the  results  with  his  or  her  superior. 
The  Army  currently  has  the  fiscal  resources  to  provide  1,500  coaching  sessions  annually,  which 
matches  demand — approximately  1  percent  of  eligible  Army  leaders  seek  out  coaching,  and 
just  shy  of  1,500  coaching  sessions  are  used  annually.  An  evaluation  of  the  coaching  service 
revealed  that  90  percent  of  the  participants  were  satisfied  with  their  coaching  experience,  and 
30  percent  sought  more  than  one  coaching  session. 

Use  and  Impact  of  MSAF  Results 

The  current  MSAF  program,  rolled  out  in  2008,  has  experienced  a  continuous  increase  in  use 
of  approximately  10  percent  per  month.  As  of  September  2014,  335,000  assessments  had  been 
conducted  under  the  program.  Data  from  a  2013  Army  evaluation  of  the  MSAF  program 
indicated  that  close  to  one-third  of  participants  list  personal  self-development  as  the  primary 
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motivation  for  completing  a  360. 2  Of  those  who  complete  the  survey,  approximately  one-half 
seek  follow-on  discussions  with  others  on  the  results.  The  most  common  form  of  follow-on 
discussion  is  with  peers  and  colleagues  (37  percent),  followed  by  discussions  with  a  supervisor 
(28  percent). 

We  are  not  aware  of  any  published  evidence  that  captures  the  efficacy  of  the  Army’s  360 
efforts  in  terms  of  effects  on  behavior  or  performance;  however,  published  studies  and  unpub¬ 
lished  data  provided  by  the  Center  for  Army  Leadership  provide  some  insights  on  participants’ 
views  about  them.  In  a  2013  program  evaluation,  Army  leaders  indicated  that  they  find  the 
program  useful  and  insightful  for  their  development.  Approximately  one-half  of  the  partici¬ 
pants  said  that  the  process  revealed  an  unknown  aspect  of  themselves,  33  percent  said  they 
discovered  a  weakness,  and  17  percent  said  they  discovered  a  strength.  Two-thirds  of  users 
indicated  intent  to  change  their  behavior  as  a  result  of  the  survey  results,  though  only  around 
10  percent  of  assessors  said  that  they  observed  behavior  change  in  individuals  who  had  gone 
through  360s.  And  two-thirds  of  the  surveyed  population  either  agreed  or  strongly  agreed  that 
the  MSAF  provides  valuable  insight. 

On  the  other  hand,  other  data  suggest  that,  while  some  might  find  360s  useful,  many 
may  not  perceive  them  to  be  as  effective  as  other  leadership  development  activities.  For  exam¬ 
ple,  in  the  Center  for  Army  Leadership’s  annual  survey  of  leaders  (Riley  et  ah,  2014),  only 
about  one-quarter  of  participants  rated  the  MSAF  as  having  “large  or  great  impact”  on  their 
development  as  leaders,  another  quarter  rated  the  MSAF  as  having  a  “moderate  impact”  on 
their  development  as  leaders,  and  50  percent  indicated  that  it  had  “small  or  no  impact.”  Out  of 
the  15  leadership  development  practices  about  which  the  survey  asked  participants,  the  MSAF 
was  among  those  with  the  lowest  perceived  impact;3  however,  as  noted  by  researchers  at  the 
Center  for  Army  Leadership,  the  MSAF  is  the  least  expensive  and  least  time-consuming  assess¬ 
ment  practice.  Both  sources  suggest  that  there  are  some  individuals  who  find  it  beneficial. 

The  most  prevalent  challenge  that  MSAF  users  face  is  obtaining  an  adequate  number 
of  individuals  to  complete  the  online  assessment.  According  to  the  2013  MSAF  evaluation 
survey,  70  percent  of  users  found  getting  enough  respondents  for  their  MSAF  to  be  the  big¬ 
gest  challenge.  Currently,  there  is  no  system  in  place  to  compel  individuals  to  complete  feed¬ 
back  forms  for  others,  and  there  is  no  system  for  ensuring  that  certain  individuals  do  not  get 
inundated  with  large  numbers  of  requests.4  This  problem  has  been  increasing  as  the  MSAF  has 
increased  in  use. 

MSAF  Costs 

The  Army  prides  itself  on  the  ability  to  execute  this  program  in  a  cost-efficient  manner  and  has 
developed  the  experience  to  maintain  and  improve  the  tool  in  house.  Recent  improvements 
have  included  a  redesign  of  the  questionnaire — shortening  the  questionnaire  by  40  percent 
and  validating  the  new  instrument — and  adding  the  capability  to  incorporate  email  addresses 


2  Data  on  the  use  of  MSAF  results  were  provided  to  RAND  during  interviews  with  personnel  at  the  Center  for  Army 
Leadership. 

3  Ranking  based  on  postsurvey  analysis.  Participants  were  not  asked  to  provide  a  relative  ranking,  only  an  assessment  of 
each  practice  independently. 

4  Completing  assessments  of  others  is  required  by  Army  Regulation  350-1  (2014).  However,  there  is  no  system  to  force 
completion. 
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across  the  entire  DoD  to  increase  the  usability  of  the  system  by  non-Army  personnel  (mostly 
as  reviewers). 

The  current  program  uses  eight  contract-year  manpower  equivalents  (CMEs)5  for  opera¬ 
tional  support  and  information  technology  support,  which  is  the  largest  cost.  An  additional 
four  CMEs  are  devoted  to  coaching.  There  is  one  full-time  program  manager,  a  GS-13,  two 
Army  majors  who  serve  full-time  as  action  officers,  and  a  GS-14  lead  scientist  (1  CME).  The 
resources  to  develop  and  pilot  the  program  are  not  included  in  these  costs  and  were  not  avail¬ 
able  to  our  research  team. 


Other  360  Efforts  in  the  Army 

The  Army  has  formally  implemented  at  least  three  other  variants  of  the  360  tailored  to  address 
specific  audiences,  contexts,  and  goals. 

The  first  is  the  Unit  360,  which  is  the  same  instrument  as  the  MSAF  but,  at  a  unit  com¬ 
mander’s  request,  is  administered  to  a  whole  unit  concurrently.  For  commanders  who  request 
a  Unit  360,  the  Army  provides  support  tools,  such  as  a  program  manager  who  can  ensure  that 
the  instrument  is  properly  distributed,  track  the  process  of  completion,  and  answer  any  ques¬ 
tions.  The  manager  has  no  access  to  the  content  provided  in  the  feedback  forms.  The  individual 
results  of  a  unit  event  are  identical  in  all  respects  to  the  MSAF  and  are  kept  strictly  confiden¬ 
tial.  However,  when  an  entire  unit  participates  in  such  an  assessment,  a  unit  roll-up  is  prepared 
that  includes  aggregate  scores  across  a  unit,  summary  reports,  and  recommendations.  The  unit, 
the  unit  commander,  and  the  commander’s  superior  officer  see  the  results,  which  can  be  used 
to  guide  training  and  development  efforts  for  the  unit. 

The  Army  also  offers  a  peer  evaluation  version  of  the  MSAF  for  the  general  officer  corps, 
in  which  general  officers  provide  feedback  on  any  officer  of  whom  they  have  sufficient  knowl¬ 
edge  to  rate.  This  feedback  is  meant  to  be  developmental,  but  it  is  reviewed  internally  for 
any  indications  of  misconduct  or  toxic  leadership — instances  have  occurred  in  which  peer 
feedback  motivated  internal  scrutiny  and,  ultimately,  disciplinary  action.  The  Army  has  also 
recently  implemented  a  360  evaluation  exclusive  to  general  officers,  which  includes  colonels  as 
well  as  other  general  officers  as  raters. 

In  October  2014,  the  Army  added  the  Commander  360  program,  which  replaces  the 
MSAF  requirement  for  lieutenant  colonels  and  colonel-level  commanders  in  the  Centralized 
Selection  Fist.  The  Commander  360  is  completed  twice  during  the  course  of  command  assign¬ 
ment,  once  within  three  to  six  months  of  assuming  command  and  again  after  15  to  18  months 
of  command.  The  Commander  360  is  similar  to  the  MSAF  in  that  it  uses  the  same  technology, 
is  based  on  the  same  leadership  doctrine,  and  is  kept  separate  from  the  formal  Officer  Evalua¬ 
tion  Reports,  but  the  questions  are  tailored  to  address  the  specific  demands  and  challenges  of 
the  command  role. 

There  are  three  other  differences  between  the  MSAF  and  the  Commander  360:  (1)  The 
commander’s  superior  (i.e.,  rater)  identifies  the  peers,  subordinates,  and  superiors  who  will 


Salaries  for  contractors  were  not  available  to  our  team,  but  a  U.S.  Office  of  Personnel  Management  white  paper  found 
contractors  to  be  cost  saving  relative  to  equivalent-level  government  employees.  In  2012,  the  approximate  base  salary  for 
a  GS-9  was  $48,000,  for  a  GS-11  was  $58,000,  for  a  GS-12  was  $70,000,  for  a  GS-13  was  $83,000,  and  for  a  GS-14  was 
$99,000  (U.S.  Office  of  Personnel  Management,  undated). 
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provide  feedback;  (2)  the  commander  and  the  rater  are  provided  the  report;  and  (3)  the  com¬ 
mander  and  his  or  her  rating  superior  must  engage  in  a  developmental  discussion  following 
the  receipt  of  the  feedback  report  to  plan  for  the  commander’s  growth  as  a  leader.  The  superi¬ 
ors  are  instructed  not  to  use  the  information  contained  in  the  360  for  evaluation.  The  Army 
relies  on  the  professionalism  of  the  senior  staff  to  consider  the  feedback  appropriately,  as  well 
as  understand  the  context  of  the  results.  The  Commander  360  program  is  designed  to  encour¬ 
age  candid  and  informed  discussions  about  leadership  and  development  between  command¬ 
ers  and  their  superiors.  The  desired  goal  is  an  active  developmental  plan  for  the  officer.  The 
Commander  360  program  was  pilot-tested  in  July  2013  and  April  2014  and  received  favorable 
reviews  from  participating  commanders  and  their  raters. 

The  Command  Climate  Survey  is  another  tool  in  the  Army  designed  to  provide  leaders 
with  views  from  below.  Although  command  climate  surveys  are  used  by  all  of  the  services,  in 
the  Army  it  is  a  supporting  piece  of  a  larger  feedback  effort.  The  Command  Climate  Survey 
captures  participants’  perspectives  on  many  aspects  of  command  (including  unit  climate, 
morale,  and  absence  or  presence  of  sexual  harassment  and  equal  opportunity  violations).  Most 
questions  do  not  focus  on  an  individual  but  rather  on  resources,  trust,  and  cooperation.  The 
influence  of  the  commanding  officer  is  reflected  in  these  answers  but  few  questions  pertain  to 
his  or  her  performance  directly.  The  results  of  these  surveys  are  publicly  shared  with  the  unit, 
the  commanding  officer,  and  the  commanding  officer’s  superior  officer.  These  surveys  are  rela¬ 
tively  short,  can  be  initiated  on  short  notice,  and  are  not  viewed  as  a  replacement  for  the  360. 


Related  Efforts 

During  our  interviews,  participants  identified  other  tools  both  in  the  Army  and  the  other  ser¬ 
vices  that  might  produce  similar  insights  to  that  of  360s — command  climate  surveys,  Inspec¬ 
tor  General  reports,  the  Marine  Corps’  Request  Mast,  and  informal  information  collection. 
Although  their  purposes  might  be  broadly  similar  to  those  of  360s,  they  do  not  solicit  the  same 
range  of  feedback  in  a  systematic  way.  Whether  these  tools  could  be  duplicative  of  the  360  ulti¬ 
mately  depends  on  how  they  are  structured  and  implemented. 

Command  Climate  Surveys 

All  the  services  administer  climate  surveys  that  commanders  periodically  use  to  assess  the 
working  environment  within  a  service  unit  and/or  command.  We  discuss  the  Army’s  use  of 
two  such  command  climate  surveys  above.  Three  interviewees  described  command  climate 
surveys  as  duplicative  of  the  360  review,  while  four  described  them  as  playing  a  complemen¬ 
tary  role. 

Regarding  the  duplicative  role,  for  example,  the  Marine  Corps’  command  climate  survey 
includes  questions  intended  to  solicit  subordinates’  perception  of  their  commanding  officers. 
Example  closed-ended  questions  ask  respondents  to  provide  agreement  with  statements  such 
as,  “My  CO  [commanding  officer]  makes  clear  what  behavior  is  acceptable  and  not  acceptable 
in  my  unit;”  “My  CO  clearly  communicates  priorities  for  the  unit;”  “My  CO  establishes  an 
environment  that  promotes  mutual  trust  and  respect.”  The  Marine  Corps’  climate  survey  also 
asks  open-ended  questions,  including  (1)  “What  is  your  unit  doing  well,  that  it  should  continue 
to  do?”  (2)  “What  is  it  not  doing  well,  that  it  should  improve  upon?”  and  (3)  “What  do  you 
know  about  your  unit  that  leadership  does  not  know,  but  should?”  The  commanding  officer 
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can,  but  is  not  required  to,  share  the  results  with  his  or  her  senior  commanding  officer  or  rater, 
as  well  as  his  or  her  team.  According  to  our  interviews,  commanders  typically  view  the  process 
as  developmental  and  are  open  to  sharing  results  with  the  senior  raters  and  subordinates. 

The  Air  Force’s  mandatory  Defense  Equal  Opportunity  Survey,  a  recent  update  of  the 
Unit  Climate  Assessment,  also  asks  questions  about  the  environment  and  people  that  could  be 
considered  duplicative  of  a  360  review.  According  to  an  interviewee,  “it  is  not  done  in  a  360 
way  bur  in  a  way  that  brings  out  stuff  that  [is]  similar  to  what  a  360  would.  ...  It  is  admin¬ 
istered  anonymously  at  each  level  [and]  focuses  on  things  like  sexual  assault-related  environ¬ 
mental  questions.” 

Similarly,  according  to  our  discussions  with  Navy  personnel,  the  360  may  be  duplicative 
of  some  elements  of  existing  Navy  command  climate  surveys.  The  Navy  representatives  with 
whom  we  met  expressed  a  preference  for  the  command  climate  survey  over  the  360  and  indi¬ 
cated  that  the  command  climate  program  is  already  useful  for  identifying  and  weeding  out 
toxic  leaders.  As  one  participant  explained,  “I  am  sure  every  senior  is  using  that  [command 
climate  survey]  to  color  their  evaluation.  ...  It  is  the  basis  for  firings  sometimes.”  According  to 
descriptions  of  the  Navy’s  current  command  climate  assessment  questionnaire  (Navy  Person¬ 
nel  Command,  2014;  Ripple,  undated),  it  aims  to  assess  several  aspects  of  organizational  effec¬ 
tiveness  and  equal  opportunity,  such  as  fair  treatment,  sexual  assault  prevention  and  response, 
favoritism,  diversity  management,  organizational  processes,  intention  to  stay,  and  job  burnout 
(Ripple,  undated).  Focus  groups  are  also  conducted  as  part  of  the  process  (Navy  Personnel 
Command,  2014),  and  example  questions  include,  “What  do  you  like  most  and  least  about 
your  command?”  “How  difficult  is  it  for  you  to  recover  from  making  a  mistake  in  the  eyes  of 
your  supervisor?”  and  “In  a  word,  how  would  you  describe  your  command?” 

Although  some  participants  in  each  of  the  services  suggested  there  could  be  overlap 
between  the  360s  and  the  climate  assessments,  the  extent  of  the  overlap  would  depend  entirely 
on  the  content  and  design  features  of  both  the  360  and  the  command  climate  assessment. 
That  is,  they  could  be  designed  in  such  a  way  as  to  capture  similar  or  different  aspects  of  per¬ 
formance.  Duplication  of  such  information  could  be  useful  in  some  contexts;  however,  to  the 
extent  that  unintentional  duplication  exists,  it  would  be  wasteful. 

Inspector  General  Reports 

All  military  personnel  have  the  right  to  file  complaints  with  the  Office  of  the  Inspector  Gen¬ 
eral.  According  to  one  interviewee,  every  year, 

there  are  thousands  of  online  complaints  that  tend  towards  ethical,  misconduct,  and  legal 
infraction  of  which  I  think  20  percent  gets  substantiated,  and,  of  those,  only  another  20 
percent  are  punitive  in  nature.  There  is  also  a  DoD  hotline  where  officers  can  call  at  any 
time  to  make  complaints  about  toxic  leadership  and  other  ethical  concerns. 

The  point  is  that  subordinates  already  have  an  avenue  through  which  to  surface  toxic  leader¬ 
ship,  and  that  avenue  is  actively  used. 

Marine  Corps  Request  Mast 

Every  Marine  is  entitled  to  participate  in  the  Request  Mast  procedure,  whereby  he  or  she  can 
ask  to  speak  with  a  superior  commander,  commanding  general,  or  inspector  general  about 
an  issue.  In  issuing  the  request,  the  officer  does  not  have  to  disclose  the  issue  he  or  she  wants 
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to  discuss,  which  allows  officers  the  opportunity  to  anonymously  express  concerns  to  senior 
leadership. 

informal  Sensing 

Interviewees  from  all  the  services  agree  that  there  are  informal  ways  of  soliciting  subordinate 
input  and  that  good  senior  leaders  already  do  so.  Yet  they  disagreed  on  whether  informal  solici¬ 
tation  and  feedback  gained  through  360  reviews  are  duplicative.  One  interviewee  felt  that  they 
might  be  duplicative  because  “there  are  no  secrets  on  a  ship,  so  I  don’t  know  that  such  a  survey 
tool  used  for  the  purposes  of  evaluation  is  necessary  to  do  that.”  This  interviewee  further  sup¬ 
ported  this  view  by  saying  that  if  a  supervisor  is  unable  to  do  this,  then  the  supervisor  should 
be  held  accountable  for  his  or  her  subordinates’  actions. 

In  comparison,  another  interviewee  shared  the  perspective  that  formalizing  the  feedback 
makes  the  difference:  “Theoretically,  there  is  nothing  in  there  that  supervisors  couldn’t  get  by 
just  going  out  and  talking  to  your  subordinates,  except  that  it  is  now  formal  and  on  paper.” 
Another  expressed  the  similar  view,  that 

walking  around  in  an  organization,  you  can  get  a  sense  of  its  health,  but  360  is  a  way  to 
get  more  granularities.  It  is  an  early-detection  tool.  We  don’t  use  360s  to  detect  criminal 
behavior,  we  want  to  use  360  as  an  early-detection  tool,  get  a  sense  of  the  organization  or 
personal  qualities  or  characteristics  [of  the  leader].” 


CHAPTER  FIVE 


Using  360  Feedback:  Evaluation  Versus  Development 


This  chapter  and  the  next  explore  answers  to  the  two  remaining  research  questions  posed  in 
Chapter  One:  Would  it  be  advisable  to  implement  360-degree  assessment  for  developmental  or 
evaluation  purposes  in  the  military ?  And  If  DoD  were  to  implement  360s,  what  are  the  imple¬ 
mentation  challenges  that  the  services  should  be  aware  of  to  ensure  success l  Although  each  chapter 
focuses  on  a  separate  research  question,  the  answers  to  both  questions  are  inextricably  inter¬ 
twined.  When  we  presented  these  questions  to  our  DoD  participants,  they  often  discussed 
implementation  challenges  at  the  same  time  as  they  debated  whether  360s  should  be  used  for 
development  versus  evaluation  purposes. 

Nevertheless,  we  chose  to  present  these  two  questions  independently,  beginning  in  this 
chapter  with  a  discussion  of  the  issues  most  hotly  debated  by  our  interviewees  in  responding 
to  the  question  of  whether  360s  should  be  used  for  development  versus  evaluation  purposes. 
In  Chapter  Six,  we  present  other  implementation  concerns  raised  by  our  participants  about  the 
use  of  360s  in  the  military — many  of  which  apply  to  the  use  of  360s  for  either  development  or 
evaluation.  Each  chapter  includes  a  summary  of  the  insights  gained  during  our  interviews  and 
a  discussion  of  key  points  of  overlap  between  the  DoD  perspectives  and  the  extant  research 
literature  and  best-practice  approaches  to  360s. 


DoD  Perspectives  on  the  Use  of  360s  for  Evaluation 

Do  military  stakeholders  believe  that  it  is  advisable  to  implement  360s  for  developmental 
or  evaluation  purposes  in  the  military?  The  short  answer  is  “no”  for  evaluation  and  “yes”  for 
development. 

Participants  clearly  see  many  more  negatives  than  positives  when  it  comes  to  using  360s 
for  evaluation  purposes,  as  shown  in  Table  5.1.  Moreover,  the  negative  sentiments  were  spread 
fairly  evenly  across  all  of  the  service  groups  with  which  we  met  (see  Table  5.2).  Even  representa¬ 
tives  from  the  Army,  which  is  already  administering  the  MSAF  to  all  officers  for  developmental 
purposes,  cited  many  reasons  for  not  implementing  the  tool  for  evaluation. 

Those  who  were  supportive  of  using  360s  for  evaluative  purposes  generally  focused  on 
using  them  for  service  members  in  leadership  positions.  Proponents  of  the  use  of  360s  for  eval¬ 
uation  believe  that  subordinate  input  is  essential  to  selecting  effective  leaders.  One  even  went 
so  far  as  to  say  that  selection  through  evaluative  use  of  the  360  may  be  of  greater  value  than  its 
development  use,  though  both  are  valuable. 

Participants  offered  up  numerous  explanations  for  why  they  believe  that  the  negatives 
outweighed  the  positives  in  using  360s  for  evaluation  purposes.  Among  the  most  strongly 


33 


34  360-Degree  Assessments:  Are  They  the  Right  Tool  for  the  U.S.  Military? 


Table  5.1 

Comments  About  Using  360s  for  Evaluation  Purposes 


Negative  Comments 

Positive  Comments 

Incorrect  data  or  dependent  on  who  responds 

15 

Might  help  with  toxic  leadership 

7 

Rater  dishonesty  problems 

12 

Might  improve  leadership 

3 

General  concern  about  use  of  360  data  to  inform 
promotions 

12 

Might  help  with  ethics  violations  or  misconduct 

4 

Concern  that  there  might  be  incorrect  or  harmful 
use  of  results 

12 

Might  improve  performance 

1 

There  would  be  too  much  cultural  resistance 

11 

Concern  that  using  for  evaluative  purposes  will 
ruin  developmental  use 

11 

Administrative  and  time  cost  is  wasteful 

9 

"Popularity  contest"  conflicts  with  mission 
objectives 

9 

Would  cause  legal  issues  for  promotions 

7 

Financial  cost 

6 

Might  duplicate  existing  efforts  (i.e.,  climate 
surveys) 

6 

Need  further  research  to  determine  efficacy  and 
cost/benefit 

2 

NOTE:  Throughout  the  interviews,  participants  described  various  pros  and  cons  and  problems  and  obstacles  to 
360s  for  evaluation  purposes.  This  table  illustrates  the  range  of  the  negative  and  positive  comments  provided. 
Each  interview  is  counted  as  only  one  comment  regardless  of  how  many  people  were  participating  in  the 
interview  or  how  many  times  the  idea  was  repeated. 


argued  reasons  were  that  they  could  cause  leaders  to  be  unwilling  to  discipline  their  subor¬ 
dinates  and  that  the  raters  would  no  longer  be  willing  to  be  honest.  But  perhaps  the  biggest 
concern  with  using  360s  for  evaluation  purposes  is  that  they  could  ruin  the  value  of  360s  for 
development.  All  of  these  are  discussed  further  in  the  next  chapter. 

If  360s  Were  Used  for  Evaluation,  Certain  Ways  Would  Be  More  Advisable 

Even  the  interviewees  who  supported  using  360s  for  evaluation  still  suggested  that  they  be 
used  only  under  certain  circumstances  and  conditions.  Others  who  were  opposed  to  their  use 
for  evaluation  suggested  that,  if  they  were  to  be  implemented,  certain  issues  would  need  to  be 
clearly  addressed.  The  sections  that  follow  provide  greater  detail  on  these  issues. 

Using  360s  to  Inform  Leadership  Selection  Decisions 

Although  our  participants  overwhelmingly  oppose  using  360s  for  evaluation,  some  did 
acknowledge  that  having  a  way  to  evaluate  someone’s  performance  from  a  range  of  viewpoints 
would  be  useful  in  making  command  or  leadership  selection  decisions. 
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Table  5.2 

Negative  Comments  About  Evaluative  Use,  by  Service  Group 


Comment 

Air  Force 

Army 

Marine  Corps 

Navy 

Joint 

Incorrect  data  or  dependent  on  who 
responds 

X 

X 

X 

X 

Rater  dishonesty  problems 

X 

X 

X 

X 

General  concern  about  use  of  360 
data  to  inform  promotions 

X 

X 

X 

X 

X 

Concern  that  there  might  be  incorrect 
or  harmful  use  of  results 

X 

X 

X 

X 

There  would  be  too  much  cultural 
resistance 

X 

X 

X 

X 

X 

Concern  that  using  for  evaluative 
purposes  will  ruin  developmental  use 

X 

X 

X 

X 

Administrative  and  time  cost 

X 

X 

X 

X 

X 

"Popularity  contest"  conflicts  with 
mission  objectives 

X 

X 

X 

X 

Would  cause  legal  issues  for 
promotions 

X 

X 

X 

Financial  cost 

X 

X 

X 

X 

Might  duplicate  existing  efforts 
(i.e.,  climate  surveys) 

X 

X 

X 

X 

Need  for  further  research  to 
determine  efficacy  and  cost/benefit 

X 

X 

They  believed  that  collecting  data  that  could  provide  a  more  complete  view  of  candidates’ 
performance  and  making  such  data  available  would  be  beneficial  when  one  reaches  a  certain 
level  of  command: 

The  only  way  we  eliminate  narcissism  from  senior  executives  is  if  we  pick  senior  execu¬ 
tives  that  have  some  insight  into  how  they  behave  towards  their  subordinates.  Most  of  the 
tools  that  we  currently  use  for  selecting  and  assigning  people  are  devoid  of  measures  of  the 
impact  that  leaders  have  on  subordinates;  how  they  generate  enthusiasm,  trust,  and  respect; 
and  how  they  care  for  their  subordinates.  This  is  such  a  delicate  area  that  I  am  certain  that 
we  need  multiple  assessments  for  selection. 

One  suggested  that  it  could  be  useful  to  include  360s  as  part  of  a  larger  commander 
evaluation  suite  that  could  include  assessment  centers  (which  employ  a  series  of  leadership 
and  workplace  simulation  activities),  command  climate  surveys,  and  the  Army’s  Commanders 
360.  Another  suggested  that  it  could  be  useful  for  everyone  in  a  command  position,  includ¬ 
ing  company  command,  battalion,  and  brigade  to  platoon  leaders.  Other  proponents  recom¬ 
mended  their  use  only  above  certain  ranks  (such  as  0-5/0-6)  because  of  the  overall  potential 
for  impact,  resource  constraints,  and  too  much  movement  among  subordinates  below  those 
ranks. 
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Even  though  there  were  a  few  proponents,  most  of  them  still  felt  that  360s  would  be 
useful  only  if  they  were  implemented  in  certain  ways  and  under  certain  conditions.  One  said, 
if  used  for  evaluative  purposes,  360s  should 

explicitly  ask  questions  that  are  geared  towards  identifying  problems.  The  surveys  should 
not  have  questions  where  you  rate  someone  on  a  scale  of  1  to  5  and  hope  that  somewhere  in 
there  you  will  identify  a  trend.  I  think  we  need  to  ask  people — to  the  tune  of,  “In  the  last 
year,  have  you  observed  your  commander  violating  any  ethical  rules  that  you  are  aware  of?” 

Another  participant  suggested  that  if  information  could  be  retained  over  the  course  of 
a  career,  perhaps  it  could  be  shared  at  the  point  when  command  and  general  officer  selection 
decisions  were  being  made.  Another  suggested  that  if  subordinate  feedback  were  solicited  for 
selection  boards,  it  should  not  be  called  360  so  as  to  distance  it  from  the  developmental  value 
of  the  process. 

As  an  alternative,  another  person  suggested  a  process  similar  to  the  background  inter¬ 
views  used  for  security  clearance  investigations — a  process  in  which  trained  interviewers  inter¬ 
view  past  subordinates  and  peers  confidentially.1  The  data  collected  during  these  interviews 
could  be  considered  when  evaluating  the  leadership  and  character  of  potential  senior  leaders 
prior  to  selection  to  high-level  positions  (command,  colonel,  or  general  officer  level),  thus  offer¬ 
ing  a  confidential  bottom-up  look  for  selection  decisions.  Choosing  this  approach  for  selection 
would  be  less  likely  to  taint  the  value  of  the  360  process  for  development  purposes,  because  the 
format  would  be  entirely  different.  It  also  might  create  an  environment  in  which  subordinates 
would  be  comfortable  providing  honest  input  and  allow  the  interviewer  to  provide  the  much- 
needed  context  for  such  information,  thus  addressing  two  major  concerns  with  using  360s  for 
evaluation  purposes. 

360-Feedback  Reports  Could  Be  Used  to  Inform  Supervisor  Ratings 

Permitting  360  information  to  be  shared  with  a  supervisor  and  allowing  that  supervisor  to 
review  the  information  and  decide  whether  and  how  to  incorporate  it  into  someone’s  perfor¬ 
mance  evaluation  was  explored  in  some  of  our  interviews.  This  seems  to  be  the  most  benign 
way  of  incorporating  360s  into  evaluations,  and  probably  the  only  way  that  key  stakehold¬ 
ers  in  the  services  would  not  reject  the  idea  outright.  However,  most  participants  were  still 
strongly  opposed  to  this  use,  again  because  of  concerns  about  it  ruining  the  use  of  360  tools 
for  development. 

Several  participants  pointed  out  that  good  supervisors  already  seek  feedback  informally 
from  subordinates  for  someone’s  formal  evaluation,  so  gaining  information  via  360  feed¬ 
back  would  be  no  different.  On  the  other  hand,  360  provides  a  more  structured  and  granular 
assessment  that  is  not  available  to  supervisors  who  are  seeking  this  information  informally. 
A  hybrid  approach,  in  which  feedback  is  provided  to  the  supervisor  and  the  ratee  long  before 
the  evaluation  period,  is  another  approach  that  allows  the  supervisor  and  ratee  to  set  goals  for 
improvement  together.  During  the  ratee’s  formal  evaluation,  the  supervisor  could  then  evalu¬ 
ate  whether  the  ratee  met  those  goals.  If  the  subordinate  made  improvements  by  the  time  that 
evaluations  were  conducted,  that  should  reflect  positively  on  the  ratee  in  performance  evalu¬ 
ations.  If  improvements  were  not  made,  the  supervisor  could  take  that  into  consideration  as 
well. 


i 


For  more  information  about  this  proposed  alternative,  see  Taylor,  2014. 
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Context  Would  Need  to  Be  Provided  If  Used  by  Promotion  Boards 

Even  those  who  support  the  use  of  360s  for  evaluation  purposes  agree  that  contextual  under¬ 
standing  is  necessary  to  properly  interpret  360  results.  Supervisors  would  likely  have  to  be 
consulted  to  help  provide  that  context.  For  example,  in  some  jobs,  there  are  few  subordinates. 
Some  assignments  are  tough  assignments  in  which  subordinates  are  going  to  dislike  anyone  in 
a  command  position.  In  some  cases,  subordinates  will  have  little  opportunity  to  observe  some¬ 
one’s  performance.  In  other  cases,  they  may  see  only  one  side  of  it.  In  some  jobs,  the  position 
may  be  extremely  challenging.  In  others,  it  could  be  relatively  unchallenging.  Sometimes,  a 
peer  or  subordinate  might  have  a  personal  agenda  or  provide  information  that  might  be  entirely 
false.  Additionally,  the  type  of  leadership  needed  changes  as  service  members  move  from  posi¬ 
tion  to  position  and  climb  the  ranks.  In  a  lower-level  position,  someone’s  performance  might 
be  viewed  as  adequate  for  that  level,  but  at  higher  levels,  it  might  not. 

Given  the  myriad  of  context  differences  that  could  occur,  even  large  differences  in  360 
ratings  about  two  different  individuals  could  be  meaningless.  Without  some  way  to  account 
for  the  context,  the  score  differences  could  be  easily  misunderstood.  Several  participants  com¬ 
mented  about  just  this  issue: 

There  is  a  balance  [that  needs  to  be  achieved]  between  giving  everybody  an  opportunity  to 
provide  feedback  and  getting  information  that’s  not  correct.  For  developmental,  this  is  not 
a  threat.  You  can  give  the  information  back  to  the  person  and  to  [his  or  her]  supervisor  to 
explain  or  make  sense  of  it.  But  for  evaluations  it  would  be  hard.  .  .  .  You  are  just  getting 
raw  data. 

One  of  the  benefits  of  being  in  the  chain  of  command  when  you  evaluate  somebody’s  360 
is  that  you  have  context  for  the  information.  If  something  you  find  raises  a  red  flag  and  you 
don’t  have  context,  you  may  draw  the  wrong  conclusion  from  the  comments. 

The  hard  ratings  that  the  ratee  receives  for  doing  what  the  service  needed  them  to  do  would 
not  accurately  capture  the  context.  Some  people  come  in  and  keep  a  ship  on  course,  and 
some  people  come  in  and  have  to  shift  things.  This  person  is  at  a  disadvantage  and  maybe 
this  story  doesn’t  get  captured  when  you  are  just  looking  at  the  numbers. 

As  noted  by  participants,  if  implemented  for  evaluation  and  used  by  promotion  boards, 
such  contextual  differences  would  have  to  be  explained.  That  would  likely  require  significant 
input  from  a  knowledgeable  supervisor,  and  supervisors  would  need  to  be  well  trained  in  how 
to  provide  that  context.2 


2  Although  supervisor  ratings  might  also  suffer  from  such  context  issues,  the  implicit  assumption  is  that  supervisors  are 
better  equipped  to  understand  the  context  than  subordinates  because  they  have  a  broader  view  of  the  organization  as  a 
whole  and  performance  of  the  individual  than  others  have.  Given  that,  in  the  military,  subordinates  are  typically  exposed  to 
only  a  few  superiors,  whereas  a  supervisor  has  a  view  of  many  subordinates  for  comparison,  it  seems  reasonable  to  conclude 
that  supervisors  would  be  better  equipped  to  provide  the  needed  context  in  an  individual’s  performance  evaluation.  Nev¬ 
ertheless,  the  recognition  by  many  participants  that  even  the  supervisor  would  need  training  in  how  to  provide  the  needed 
context  is  an  acknowledgment  that  supervisor  input  itself  can  vary  in  quality  and  accuracy. 
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DoD  Perspectives  on  the  Use  of  360s  for  Development 

In  contrast  to  the  largely  negative  comments  about  using  360s  for  evaluation  purposes,  inter¬ 
viewees  expressed  many  more  positive  comments  about  developmental  use  of  360s  (Table  5.3). 
They  also  expressed  some  concerns  (also  shown  in  Table  5.3),  but  the  issues  were  fewer  than 
those  expressed  about  evaluative  use.  Additionally,  participants  reacted  far  more  positively  to 
the  idea  of  considering  360s  for  developmental  purposes,  even  in  light  of  the  negatives  they 
mentioned. 

A  number  of  those  interviewed  talked  about  why  360s  have  value  when  used  for  devel¬ 
opment.  They  believe  that  360s  help  identify  and  promote  positive  traits,  as  well  as  highlight 
weaknesses,  which  gives  officers  an  opportunity  to  strengthen  those  areas.  In  general,  officers 
want  to  understand  their  strengths  and  weaknesses,  and  360s  are  an  opportunity  to  gain 
insight — a  tool  that  can  facilitate  “conversation  between  leaders  and  subordinates  about  what 
is  needed  to  develop.” 

When  I  did  it,  I  sought  out  people  who  will  have  nothing  to  gain  by  telling  me  I  was  a 
great  guy.  I  wanted  to  find  out  where  I  could  grow.  You  take  the  survey  yourself  and  other 
people  grade  you.  In  my  experience,  I  graded  myself  harder  than  everybody  else.  And  in 
the  military,  that  tends  to  be  common.  In  the  commercial  world,  it  is  the  opposite.  Your 
typical  military  guy  wants  to  grow.  The  temptation  to  whitewash  things  may  happen  in  the 
[evaluation]  side  but  not  in  the  development  side. 

[We  get]  tremendous  feedback  on  the  ones  we  do.  Everyone  treasures  it.  I  would  say  for 
85  percent  who  do  a  360,  it’s  the  first  one  they  ever  had.  And  when  I  use  words  like,  “Hey, 


Table  5.3 

Comments  About  Using  360s  for  Developmental  Purposes 


Negative  Comments 

Positive  Comments 

Administrative  and  time  cost 

8 

Might  improve  leadership 

13 

Need  for  further  research  to  determine  efficacy 
and  cost/benefit 

7 

Might  help  with  toxic  leadership 

12 

Concern  that  superiors  will  use  developmental 
results  in  an  evaluative  way 

7 

Might  improve  performance 

10 

Concern  that  there  might  be  incorrect  or  harmful 
use  of  results 

7 

Raters  might  be  honest 

8 

Incorrect  data  or  dependent  on  who  responds 

5 

Might  help  with  ethics  violations 

7 

Financial  cost 

5 

Might  help  with  misconduct 

7 

Rater  dishonesty  problems 

3 

The  service  would  be  culturally  receptive  to  it 

7 

There  would  be  too  much  cultural  resistance 

3 

"Popularity  contest"  conflicts  with  mission 
objectives 

3 

NOTE:  Throughout  the  interviews,  participants  described  various  pros  and  cons  and  problems  and  obstacles 
to  360s  for  developmental  purposes.  This  table  illustrates  the  range  of  the  negative  and  positive  comments 
provided.  Each  comment  is  counted  only  once  per  interview,  regardless  of  how  many  people  were  participating 
in  the  interview  or  how  many  times  the  idea  was  repeated  in  the  interview. 
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did  it  help  you  identify  some  blind  spots  or  gaps?”  or  “Wow,  I  didn’t  really  know  I  came 
across  that  way,”  .  .  .  they  all  [say],  “You  bet;  that’s  exactly  what  happened.” 

Proponents  of  360  assessments  believe  that  officers  can  grow  from  an  approach  that  pro¬ 
vides  different  perspectives  on  how  attitudes  and  behaviors  are  perceived  by  others: 

It  helps  people  to  see  gaps  that  they  have.  We  may  all  perceive  that  we  all  have  certain  lead¬ 
ership  traits  and  personalities  that  are  more  effective  than  they  actually  are.  They  may  speak 
to  a  certain  group  but  not  to  another  group.  So  [the  feedback]  helps  you  focus  on  things 
that  [you]  do  or  say  that  I  think  is  very  open  but  ends  up  threatening  people.  It  helps  you 
see  how  you  could  be  a  more  effective  leader,  certainly  on  the  developmental  side. 

The  character  piece  is  a  blind  spot  for  folks  [that  the  360  can  address]. 

Unlike  evaluative  use,  there  were  some  differences  across  services  in  the  number  of  nega¬ 
tives  that  were  raised  in  using  360s  for  developmental  use.  Interviewees  from  the  Army  tended 
to  express  the  fewest  negative  comments  about  using  the  360  for  development,  and  the  greatest 
number  of  positive  comments,  although  they  too  recognized  that  there  are  limits  to  its  use¬ 
fulness.  The  Air  Force  and  the  Joint  Staff  also  had  many  positive  comments  about  its  use  for 
development.  The  Navy  and  Marine  Corps  representatives  had  positive  things  to  say  as  well, 
although  they  were  the  most  reserved  in  their  support  for  use  of  the  tool. 


Overall  Conclusions  on  Advisability  of  Evaluation  Versus  Development 

From  the  summary  of  the  comments  provided  above,  it  is  clear  that  most  stakeholders  believe 
that  the  disadvantages  of  using  360s  for  evaluation  purposes  outweigh  the  advantages.  Even 
the  Army — the  organization  that  cited  the  fewest  downsides  to  developmental  360s — cites 
many  reasons  it  could  be  detrimental  and  ill-advised  as  an  evaluation  tool.  Although  these 
results  alone  would  suggest  that  our  participants  would  argue  against  using  360s  for  evaluation 
in  the  military,  we  also  asked  participants  directly  whether  they  would  be  supportive  of  using 
360s  for  each  purpose  (development  or  evaluation).  A  breakdown  of  their  answers  is  shown  in 
Table  5.4. 

Stakeholders  were  overwhelmingly  against  using  the  tool  for  evaluation.  Fourteen  out  of 
the  19  interview  groups  provided  us  with  an  emphatic  “no”  when  asked  whether  using  360s 
for  evaluations  was  advisable.  Although  most  participants  provided  us  with  a  clear  yes  or  no 
answer  when  asked  directly,  two  did  not.  One  explained  that  developmental  uses  should  be  in 


Table  5.4 

Number  of  Interview  Groups  That  Believe  It  Is  Advisable  to 
Use  360s  for  Evaluation  Versus  Development 


Response 

Development 

Evaluation 

Yes 

16 

3 

It  depends 

3 

0 

No  clear  answer 

0 

2 

No 

0 

14 
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place  first  before  the  services  even  consider  exploring  using  360s  for  other  purposes.  The  other 
stated,  “It  would  take  some  time  and  study,”  and  either  way,  “the  purposes  [developmental  and 
evaluation]  should  never  be  merged.  If  you  mix  the  two,  it  will  be  problematic.”  Three  inter¬ 
view  groups  responded  “yes,”  indicating  that  they  would  consider  using  360s  in  evaluations 
but  that  many  safeguards  would  need  to  be  in  place — that  360s  should  be  used  only  for  certain 
groups  and/or  under  certain  circumstances  and  that  more  research  would  be  needed  to  explore 
consequences  of  their  use  over  time.  Some  of  the  conditions  under  which  they  could  envision 
using  360s  for  evaluation  purposes  are  described  in  the  next  section. 

In  contrast,  16  of  the  19  interview  groups  expressed  clear  support  for  using  developmental 
360s  for  a  subset  of  the  officer  population.  The  remaining  three  provided  slightly  more  cautious 
responses.  Two  of  these  recommended  that  they  be  used  for  development  but  only  on  a  vol¬ 
untary  basis.  Of  those  two,  one  recommended  that  they  be  used  only  with  more  information 
about  the  cost-benefit  ratio  before  a  full-scale  rollout,  and  the  other  felt  that  “if  it  is  voluntary, 
then  the  people  [who]  least  need  it  will  do  it  and  the  ones  [who]  need  it  [won’t]  do  it.”  Of  the 
16  groups  that  clearly  recommended  using  360s  for  developmental  purposes,  12  recommended 
them  only  for  those  at  the  0-4  or  0-5  ranks  and  higher.  Most  cited  resource  constraints  as 
the  rationale  for  limiting  360  use  to  those  in  higher  grades.  This  topic  is  covered  in  greater 
detail  in  the  following  chapter.  Only  two  interviewees,  both  from  the  Army,  recommend  using 
the  360  for  development  for  all  military  personnel,  one  describing  it  as  “totally  worth  it  at  all 
levels,”  and  the  other  as  being  useful  for  “everyone,  with  openness  between  ratee,  superior,  and 
subordinate.” 

Not  Ideal  for  Addressing  Toxic  Leadership,  Ethics  Violations,  and  Other  Problem  Behaviors 

In  addition  to  asking  participants  whether  360s  should  be  used  for  evaluation  or  for  develop¬ 
ment  purposes,  we  specifically  asked  participants  whether  they  thought  360s  would  be  useful 
in  addressing  toxic  leadership  or  other  problem  behaviors — whether  as  a  development  or  evalu¬ 
ation  tool.  A  majority  felt  that  eliminating  toxic  leadership  was  an  important  goal  but  that 
360s  were  not  the  best  tool  for  accomplishing  it.  Instead,  they  pointed  to  other  tools  already  in 
place  that  are  specifically  designed  to  address  that  need: 

There  are  all  sorts  of  methods — open-door  policy,  chaplain,  observation,  complaining, .  .  . 
etc.  It  is  a  brigade  commander’s  responsibility  to  know  whether  his  [or  her]  battalion  com¬ 
manders  are  toxic.  By  observing  and  talking  to  soldiers  two  levels  down,  you  can  find  out 
what  you  need  to  know  about  the  quality  of  the  leaders  under  you. 

There  are  already  tools  in  place  .  .  .  [such  as  the  inspector  general  process]  to  find  out  the 
environment.  Toxic  leaders  are  being  found  this  way. 

There’s  also  mandatory  assessments,  such  as  the  Defense  Equal  Opportunity  [Climate] 
Survey  .  .  .  which  asks  about  the  environment  and  people.  It  is  not  done  in  a  360  way  but 
in  a  way  that  brings  out  stuff.  It  is  administered  anonymously  at  each  level.  It  focuses  on 
things  like  sexual  assault-related  environmental  questions. 

Although  they  did  not  feel  that  addressing  toxic  leadership  was  the  main  purpose  of  360s, 
participants  did  note  that  some  toxic  leaders  might  be  improved  with  360s.  Participants  dis¬ 
cussed  three  types  of  toxic  leaders.  The  first  is  the  toxic  leader  who  means  well  but  is  unaware 
of  how  his  or  her  actions  are  viewed  by  others  or  is  unable  to  improve  on  his  or  her  own.  For 
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those  toxic  leaders,  360s  for  developmental  purposes  might  open  their  eyes  to  their  actions  and 
ultimately  have  a  positive  impact  on  their  behavior.  The  second  type  is  the  toxic  leader  who 
has  no  desire  to  change.  Participants  agreed  that  many  toxic  leaders  are  of  this  ilk,  and,  for 
them,  360s  for  developmental  purposes  would  likely  have  no  effect.  They  also  noted  that,  if 
360s  were  included  in  evaluations,  they  might  have  an  impact  on  these  people  (because  evalu¬ 
ations  are  high-stakes),  but  these  individuals  are  so  rare  that  using  360s  in  this  way  would  be 
an  expensive  and  time-consuming  way  to  find  them.  Participants  again  pointed  to  other  ways 
of  finding  these  people  that  would  be  much  more  cost-effective  (such  as  through  anonymous 
reporting  channels,  climate  surveys,  informal  discussion,  or  inspector  general  complaints).  The 
third  type  of  toxic  leader  described  is  generally  viewed  negatively  because  he  or  she  makes 
tough  decisions  that  subordinates  dislike.  This  individual  might  appear  toxic  on  paper,  but,  in 
actuality,  he  or  she  would  be  doing  exactly  what  was  most  needed.  Participants  believed  that 
360s  might  backfire  in  such  cases  as  this. 

Ethics  violations  and  other  problem  behaviors  were  generally  discussed  as  a  different  issue. 
For  those  types  of  behaviors,  the  belief  was  that  (1)  reporting  of  that  kind  of  behavior  would 
lead  to  a  myriad  of  legal  due-process  concerns;  (2)  ratees  would  be  no  more  likely  to  report  it 
on  a  360  than  they  would  using  the  other  avenues  available  to  them,  and  they  should  be  report¬ 
ing  it  through  those  avenues;  and  (3)  most  people  already  know  when  someone  is  engaged  in 
problem  behaviors — a  360  would  not  provide  any  new  information.  The  latter  belief — that 
everyone  already  knows — was  also  expressed  about  toxic  leadership: 

Most  people  have  worked  for  a  toxic  leader  at  some  point,  and  it  wasn’t  a  secret.  Everybody 
knew  who  was  a  jerk.  The  question  is:  What  was  done  about  it?  You  could  find  them;  then 
what?  What  did  we  do  about  it?  Doesn’t  the  culture  determine  what  happens  at  that  point? 

Do  we  need  360  or  do  we  know?  And  the  bigger  question  is  what  are  we  doing  about  it?  In 
many  cases,  alert  leaders  know,  but  would  they  do  something  about  it? 

Finally,  respondents  suggested  that  if  the  goal  of  using  360s  for  evaluation  was  to  find 
toxic  leaders  or  bad  behavior,  360s  would  create  a  policing  culture  that  would  not  only  under¬ 
mine  the  benefits  of  360s  but  also  engender  greater  paranoia  and  distrust  in  the  system: 

The  360  is  not  a  magic  wand,  ft  is  just  a  tool  for  leader  development  that  could  have  a  posi¬ 
tive  impact,  but  it  may  not.  Take,  for  example,  the  guy  who  had  an  issue  at  Fort  Bragg.  1 
don’t  know  if  he  had  a  360  or  not  and,  if  he  did,  it  may  or  may  not  have  had  an  impact 
on  the  behavior  he  displayed.  To  say  we  want  to  do  it  to  mitigate  over-the-top  behavior  of 
some  senior  officials  is  not  the  angle  we  are  coming  at  it  from.  When  people  are  senior,  over 
50  years,  they  are  the  way  they  are,  a  360  is  not  likely  to  change  them  several  folds.  The 
bottom  line  is  that  we  view  this  as  a  developmental  tool  far  more  than  [as]  a  tool  to  prevent 
behavior  that  a  small  percentage  [of  service  members]  exhibit. 
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This  chapter  addresses  the  second  of  the  two  implementation  questions:  IfDoD  were  to  imple¬ 
ment  360s,  what  are  implementation  challenges  of  which  the  services  should  be  aware  to  ensure  suc¬ 
cess?  Whether  the  services  use  360s  for  development,  evaluation,  or  some  combination  of  the 
two,  numerous  issues  would  need  to  be  ironed  out  when  implementing  this  tool.  Some  are  of  a 
practical  nature,  such  as  identifying  the  right  raters,  coaching,  and  legal  issues.  Other  concerns 
are  more  esoteric  in  nature,  yet  equally  important — such  as  how  to  use  360  feedback  without 
eroding  trust  and  how  to  ensure  acceptance  within  the  military  culture.  System  fundamentals, 
such  as  the  accuracy  of  the  ratings  and  the  logistics  required  for  implementation,  also  need  to 
be  considered.  All  of  these  topics  were  raised  during  the  course  of  our  interviews,  and  many  are 
also  echoed  by  experts  in  the  field. 


Identifying  the  Right  Raters 

As  noted  in  an  earlier  chapter,  in  practice,  some  360  feedback  tools  do  not  involve  all  three 
traditional  feedback  sources:  subordinates,  peers,  and  supervisors.  Although  some  experts  in 
360-degree  assessment  (see,  for  example,  Foster  and  Law,  2006)  argue  that  feedback  tools  that 
exclude  one  or  more  of  the  three  sources  or  that  include  other  sources  (such  as  customer  rat¬ 
ings)  should  technically  be  called  “multisource  feedback”  rather  than  360,  we  included  these 
other  forms  of  multisource  feedback  in  our  broad  definition  of  360  for  the  purposes  of  this 
review.  Our  participants  also  conceived  of  it  broadly,  offering  numerous  thoughts  about  who 
might  be  most  appropriate  as  raters  in  military  360s. 

Overall,  participants’  views  differed  on  who  should  participate  in  360s;  but  despite  the 
differences  in  perspective,  the  general  goals  of  their  comments  were  similar:  to  gather  the  right 
information  in  an  efficient  way  and  without  undue  duplication.  Several  people  with  whom  we 
spoke  believed  that  a  full  360  review  with  peers,  subordinates,  and  supervisors  all  providing 
feedback  would  not  be  necessary  or  even  beneficial  in  the  services.  Instead,  they  suggested  that 
a  partial  360  using  only  some  of  the  sources  would  be  better.  Some  felt  that  peers  and  subordi¬ 
nates  might  be  interchangeable  and  that  little  additional  insight  would  be  gained  from  spend¬ 
ing  the  resources  on  capturing  information  from  both.  Others  felt  that  peers  and  subordinates 
had  unique  perspectives.  Still  others  argued  that  the  contribution  of  peers  versus  subordinates 
would  depend  entirely  on  the  rater’s  and  ratee’s  positions. 
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Several  people  suggested  that  the  main  insight  of  the  360  is  the  subordinate  perspective, 
and  so  greater  emphasis  should  be  placed  on  that  source: 

For  boards,  for  a  variety  of  reasons  (administrative  simplicity  is  one  but  not  the  main  one), 

I  would  .  .  .  use  only  subordinate  input.  Subordinate  input  would  not  be  [from]  the  people 
[who]  are  immediately  working  with  this  person;  they  could  be  included,  but  you  should 
go  back  to  prior  commands  and  go  back  to  prior  people.  There  are  a  number  of  advantages 
in  that,  to  include  enhancing  the  anonymity  of  the  process. 

Others  felt  that  the  supervisor  already  provides  structured  feedback  and  so  tasking  the 
supervisor  with  completing  a  360  is  wasteful,  because  any  information  gathered  is  likely  to  be 
duplicative  of  the  standard  performance  evaluation. 

Some  felt  that,  in  the  typical  military  workplace  setting,  peers  have  little  opportunity  to 
observe  each  other’s  performance  and  hence  would  have  little  to  contribute.  In  some  military 
training  courses,  however,  peers  may  have  plenty  of  opportunity  to  observe  someone’s  perfor¬ 
mance  and  may  be  quite  well  situated  to  provide  appropriate  feedback.  Others  argued  that  peer 
feedback  should  be  avoided  at  higher  levels  of  command,  because  the  climate  can  become  com¬ 
petitive  and  affected  by  politics.  The  competitive  nature  of  these  positions  would  reduce  the 
value  of  feedback  provided  by  peers  such  that  the  exercise  would  not  be  worth  the  investment: 

Peer  input  can  be  useful.  But  peer  input  is  particularly  sensitive  in  competitive  places. 

And  senior  people,  it’s  a  very  delicate  thing  and  I  don’t  think  they  add  that  much.  And  the 
potential  for  friction  and  competition  and  lack  of  opportunity  to  observe — an  awful  lot  of 
0-6s  in  the  services  don’t  really  see  how  other  people  are  doing. 

Still  others  believe  that  a  full  360  (with  supervisors,  peers,  and  subordinates)  is  beneficial 
in  certain  circumstances: 

Yes,  go  for  the  full  range  in  development.  The  boss  is  already  there,  but  you  want  self- 
compared  to  boss,  peer,  subordinate.  Ffopefully  the  boss  is  evaluating  and  giving  feedback 
frequently,  but  yes,  for  development  purposes  we  need  the  true  360. 

Moreover,  as  noted  by  a  participant,  even  a  rater’s  familiarity  with  the  ratee  can  be 
important: 

Another  challenge  is  that  the  assessed  officer  will  send  it  to  someone  who  doesn’t  really 
know  [him  or  her].  If  we  are  going  to  use  it  for  evaluation,  whether  we  are  asking  the  right 
people  to  do  the  assessment  is  critical. 

Experts  generally  agree  that  the  usefulness  of  the  sources  can  vary  from  setting  to  setting, 
and  even  within  one  setting,  subordinates  or  peers  may  be  able  to  comment  about  only  a  subset 
of  performance.  One  way  to  maximize  the  usefulness  of  the  information  is  to  tailor  the  ques¬ 
tions  to  each  source  (DeNisi  and  Kluger,  2000). 

The  point  to  be  taken  away  from  these  varying  perspectives  is  that  part  of  establishing 
a  360-feedback  system  is  to  give  careful  consideration  to  who  should  participate.  In  part,  the 
answer  lies  in  the  purpose  for  the  360  and  how  the  service  intends  to  use  the  results — a  topic 
discussed  in  the  following  chapter — and  in  part  the  answer  lies  in  the  type  of  position. 
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Incorporating  Coaching  and  Follow-Up 

Many  experts  in  360s  recommend  incorporating  coaching  or  some  other  form  of  follow-up 
activity  to  help  improve  the  likelihood  that  360s  will  have  positive  impacts  (see,  for  example, 
Atwater,  Brett,  and  Charles,  2007;  and  DeNisi  and  Kluger,  2000).  Indeed,  there  is  evidence 
from  empirical  research  that  coaching  can  have  benefits  for  subsequent  job  performance.  In 
one  study,  managers  who  met  with  direct  reports  to  discuss  their  upward  feedback  improved 
more  than  managers  who  did  not,  and  they  improved  more  in  years  when  they  discussed  the 
previous  year’s  feedback  than  in  years  when  they  did  not  (Walker  and  Smither,  1999).  Review¬ 
ing  plans  and  progress  quarterly,  seeking  input  from  coworkers,  and  receiving  coaching  and 
feedback  are  strongly  related  to  improvement  in  ratings  over  time  (although  managers  engaged 
in  these  activities  relatively  infrequently)  (Hazucha,  Hezlett,  and  Schneider,  1993).  Another 
study  found  that  systematic  coaching  on  how  to  interpret  discrepancies  between  self-ratings 
and  other  ratings  improved  manager  self-awareness  and  subsequent  performance  (as  rated  by 
others),  as  well  as  attitudes  and  commitment  of  both  the  managers  and  the  other  employ¬ 
ees  (Luthans  and  Peterson,  2003).  However,  experts  acknowledge  that  coaching  itself  can  be 
tricky;  thus,  the  people  providing  the  coaching  need  to  be  skilled  in  it.  This  means  that  man¬ 
agers  or  coaches  need  to  be  trained  in  how  to  interpret  the  feedback  and  how  best  to  deliver  it, 
another  added  cost  to  implementing  360s  properly. 

The  importance  of  including  coaching  was  emphasized  repeatedly  in  our  interviews.  There 
was  near-universal  agreement  among  those  who  have  had  experience  with  360s  that  properly 
structured  coaching  is  essential  for  successful  implementation.  In  many  cases,  this  opinion  was 
shaped  by  experience  with  the  smaller  360  programs  found  throughout  the  services  that  target 
senior  officers.  In  a  summation  of  a  common  sentiment  about  coaching,  one  interviewee  said, 
“We  think  the  coaching  will  be  the  most  critical  piece  for  getting  developmental  utility.” 

Participants  also  brought  up  coaching  issues  to  consider,  including  who  should  provide 
the  coaching,  when  and  with  what  frequency,  and  the  feasibility  of  providing  coaching  to 
large  numbers  of  personnel.  The  value  of  extended  conversations  with  coaches  who  are  able  to 
interpret  the  data  and  assist  with  development  plans  was  mentioned  as  an  important  element 
for  success: 

The  people  I  have  talked  to  and  myself  when  I’ve  had  360s,  I  find  the  great  value  came  from 
having  a  great  coach  outside  of  the  organization,  who  takes  [these]  data  and  is  trained  in 
how  to  assess  [them] .  And  sits  down  in  a  very  lengthy  conversation  with  me,  often  an  hour, 
walking  through  the  results,  asking  those  probing  questions  that  in  the  end  I  say,  “hmm,  I 
have  some  blind  spots;  here  [are]  some  areas  that  I  can  maybe  improve  in.” 

How  you  follow  up  and  create  a  plan  is  magic.  There  needs  to  be  some  sort  of  plan  and 
commitment  to  a  continuation.  It  can’t  be  a  one-off.  It’s  a  developmental  tool.  What  are  you 
going  to  do  about  this  information?  What  do  you  do  to  strengthen  the  response? 

Anyone  [who]  uses  a  coach  finds  it  more  rewarding,  and  that’s  the  most  expensive  part.  And 
you  have  to  get  that  right. 
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Coaches  are  often  the  largest  expense  of  the  programs  and  are  either  retired  senior  officers 
or  professional  outside  coaches.  The  Army,  which  has  the  largest  360  program,  has  internalized 
the  coaching  function: 

The  expectation  is  that  the  rater  will  sit  down  and  mentor  the  officer.  This  is  not  the  single 
source  for  coaching.  Just  like  with  any  other  MSAF,  all  officers  still  have  access  to  profes¬ 
sional  coaches  if  they  want,  and  they  can  click  on  access  to  the  doctrine  and  virtual  sup¬ 
port.  Training  is  being  developed  to  help  the  raters  learn  how  to  be  .  .  .  better  [mentors]. 

Several  participants  also  brought  up  the  theme  of  integrating  the  feedback  into  service 
culture.  One  person  brought  up  that  integrating  feedback  evaluation  into  the  military  educa¬ 
tion  curriculum  is  valuable  to  create  comfort  with  and  commitment  to  the  system.  Another 
person  spoke  of  the  broader  cultural  change  that  feedback  can  engender: 

I  believe  that  we  have  to  move  towards  important  meaningful  dialogues  about  “how  am 
I  doing?”  Not  just  “how  am  I  doing  with  getting  the  job  done?”  which  is  important  and 
happens  now.  But  “how  am  I  doing  with  building  camaraderie  and  cohesion  and  building 
professional  relationships?”  which  requires  some  coaching  and  mentoring  and  some  frank 
discussions  with  some  officers  .... 


Addressing  Fairness  and  Legal  Concerns 

Participants  were  concerned  that  a  number  of  fairness  and  legal  concerns  might  arise  as  a  result 
of  collecting  360  information.  Most  were  associated  with  either  the  fact  that  a  rater’s  responses 
are  anonymous  or  decisions  regarding  when  something  would  trigger  an  official  investigation. 

Interviewees’  concerns  about  anonymity  were  more  notable  in  the  evaluative  context. 
When  360  ratings  are  anonymous  or  confidential  (that  is,  either  raters  cannot  be  identified  or 
their  identity  is  otherwise  protected),  a  number  of  potential  legal  implications  arise  if  confiden¬ 
tiality  concerns  impact  the  formal  evaluation  process  (either  officially  or  unofficially).  If  a  ratee 
ultimately  disagrees  with  information  in  a  360  but  it  goes  into  his  or  her  official  record,  how 
can  the  ratee  appeal?  In  a  traditional  fitness  report,  there  is  due  process,  but  for  360s  appeals 
and  official  investigations,  this  might  not  be  workable  if  confidentiality  is  to  be  maintained.  As 
one  interviewee  put  it, 

There  is  nothing  to  go  back  and  prove  them  wrong.  They’ll  never  know  who  submitted  the 
information.  With  the  appeals  process,  you  can  get  negative  information  removed  from 
your  promotions  record.  If  there  is  something  bad  there,  somebody  will  come  back  to  say 
“we  have  to  prove  this  thing.”  ...  In  the  end,  the  bad  thing  hardly  gets  reported  because 
the  anonymity  gets  lost  in  investigation. 

Our  system  of  government  gives  you  a  right  to  offer  a  rebuttal.  How  do  you  do  that  in  an 
anonymous  confidential  survey  where  the  people  filing  it  want  to  remain  anonymous? 

Several  interviewees  explained  that  anonymity  might  be  more  difficult  to  protect  when 
using  360s  for  evaluation  purposes  because  military  personnel  have  the  right  to  learn  about 
adverse  information  that  affects  their  performance  reviews  and  promotion.  Some  expressed 
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concerns  that  it  could  lead  to  investigations  intended  to  identify  the  anonymous  rater.  One 
stated, 

It  is  harder  for  us  to  keep  that  confidential  if  someone  is  asking  for  it,  if  it  is  adverse  and 
being  used  for  evaluation.  If  for  developmental,  then  as  long  we  can  show  that  it  didn’t  leak 
into  the  evaluations,  it’s  not  a  problem. 

With  respect  to  anonymity,  a  common  fairness  concern  among  interview  groups  was 
whether  anonymity  could  actually  be  preserved  when  the  pool  of  possible  raters  is  narrow: 

If  I  supervise  15  people  or  five  people  and  I  have  to  do  a  360,  when  someone  calls  out  some¬ 
thing,  I  know  exactly  who  said  it. 

There  was  also  concern  over  the  exploitation  of  anonymity.  Interview  participants  worried 
that  an  individual  could  use  anonymity  to  communicate  strong  sentiments  without  regard  for 
validity  or  accuracy  of  those  assessments.  Potential  misuses  of  the  anonymity  of  the  360  span 
concerns  over  incomplete  data,  biased  data,  and  collusion  and  sabotage: 

It  is  difficult  to  get  people  to  do  the  survey  because,  no  matter  what  they  said,  people  were 
concerned  about  the  anonymity. 

Another  concern  raised  by  our  interviewees  was  the  appropriate  course  of  action  if  some¬ 
thing  illegal  or  that  otherwise  violates  military  codes  of  conduct  is  raised  in  a  360.  For  exam¬ 
ple,  if  a  complaint  is  filed  against  an  officer,  typically  an  investigation  occurs  to  determine 
the  accuracy  of  the  information.  But,  if  the  information  is  guaranteed  to  be  anonymous  or 
confidential,  is  the  organization  even  legally  allowed  to  take  action  on  it?  As  an  interviewee 
questioned, 

Who,  if  anyone,  is  supposed  to  check  for  such  information?  Do  we  keep  quiet  because  it  is 
anonymous?  Do  we  do  an  investigation  or  not?  I’m  not  sure.  .  .  . 


Maintaining  Trust  Within  the  Organization 

Discussions  of  360s  often  mention  trust  in  several  ways.  Some  described  general  concerns  that 
360s  would  lead  to  distrust  of  one’s  peers  and  subordinates  and  the  organization.  The  armed 
services  see  trust  as  an  essential  feature  of  mission  preparedness,  and  the  impact  of  the  360 
is  a  high  concern.  In  our  interviews,  we  found  a  range  of  opinions  on  this  topic.  The  greater 
portion  of  our  participants  was  concerned  that  360s — particularly  for  evaluation  purposes — 
could  erode  trust  that  is  needed  for  units  to  operate  successfully.  There  is  concern  about  what 
360s  “might  do  to  the  cultures  of  teamwork  and  competence  [and  about]  what  the  impact  will 
be  on  those.”  There  are  also  concerns  that  360s  might  be  perceived  as  negatively  affecting  ser¬ 
vice  members’  careers — and  in  doing  so  undermine  trust  in  the  broadest  sense: 

A  key  concern  is  the  impact  on  organizational  trust:  How  do  you  foster  cohesion  in  unit? 

Our  command  climate  tells  us  we  could  do  a  better  job  of  building  trust.  An  assessment 
360  has  the  potential  to  cause  people  to  cover  up  rather  than  open  up.  We  depend  on  trust, 
and  anything  that  breaks  that  down  is  not  good. 
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Right  now  the  services  are  downsizing;  putting  in  one  more  component  that  could  under¬ 
mine  a  career  will  increase  the  paranoia  about  whether  a  service  member  is  going  to  have 
a  job  tomorrow. 

Others  felt  that  the  tool  could  be  effectively  used  in  higher  ranks  without  issue  but  that 
service  members  at  more-junior  grades  of  lieutenant  and  sergeant  might  not  have  the  maturity 
to  handle  the  tool  without  adversely  affecting  trust: 

1  think  the  possibility  of  creating  an  environment  where  day-to-day  mutual  trust  and 
respect  is  somewhat  compromised  by  this  .  .  .  when  we  are  talking  about  people  at  that 
level.  1  have  used,  and  1  think  it’s  good  to  use,  these  instruments  at  the  very  basic  level  if 
it  is  done  in  a  proper  environment.  But  basically  I  am  not  sure  that  company  commanders 
and  [their]  lieutenants  and  sergeants  and  so  forth  are  quite  ready  to  handle  it.  This  is  my 
secondary  concern. 

And  still  others  felt  that  the  360  can  foster  greater  transparency  and  lead  to  an  overall 
improvement  in  trust: 

They  could  be  useful  for  establishing  greater  trust.  The  Army  has  a  unit  360,  and  I’m  really 
impressed  with  the  results.  It  strikes  me  as  really  sharp.  It  starts  from  the  developmental 
perspective  and  can  improve  transparency  in  the  unit. 

Another  topic  raised  by  participants  and  experts  alike  (for  example,  DeNisi  and  Kluger, 
2000)  related  to  the  importance  of  trust  associated  with  the  confidentiality  of  the  feedback 
itself.  Without  confidentiality  of  the  results,  raters  might  not  be  honest,  and  ratees  might  not 
be  motivated  to  use  the  information  in  constructive  ways: 

Confidentiality  rules  will  make  or  break  the  developmental  effort.  The  subordinates  are 
not  going  to  give  you  and  me  accurate,  candid  feedback  if  they  think  [they]  are  going  to  be 
known.  It’s  got  to  be  sure  that  the  boss  does  not  know  who  we  are. 

Confidentiality  and  debriefs  by  certified  coaches  are  the  centerpieces  of  the  development 
approach.  This  confidential  feedback  (1)  creates  a  supportive  and  trusting  environment 
for  the  growth  and  development  of  its  officers;  (2)  promotes  confidence  in  the  results; 

(3)  ensures  those  providing  responses  retain  their  confidentiality  without  fear  of  attribu¬ 
tion;  and  (4)  provides  the  officer  an  insightful  view  of  what  others  see. 

In  the  context  of  confidentiality  and  maintaining  trust,  many  wondered  whether  super¬ 
visors  should  be  given  access  to  their  subordinates’  360  information  for  purposes  of  coaching 
and  mentoring.  This  scenario  had  critics,  but  there  was  generally  less  resistance  to  sharing  360 
results  with  supervisors  for  coaching  and  mentoring  than  for  evaluation  purposes.  Concerns 
included  that  the  supervisor  might  not  be  able  to  set  aside  the  information  when  it  came  time 
to  provide  annual  performance  evaluation  comments.  Others  suggested  that  it  might  lead 
supervisors  to  feel  as  though  their  hands  were  tied  because  they  could  not  take  official  action 
about  negative  information.  People  also  expressed  concerns  that  raters  would  water  down  com¬ 
ments  if  the  information  would  be  shared  with  supervisors,  or  people  would  pick  only  raters 
who  would  say  positive  things,  negating  the  developmental  benefit  of  honest  feedback. 
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Even  participants  from  the  Army  were  also  generally  cautious  about  sharing  information 
with  supervisors,  even  for  developmental  purposes.  The  Army’s  Commander  360  is  the  Army’s 
first  attempt  at  sharing  360s  with  supervisors.  Although  the  intent  is  that  supervisors  use  the 
information  to  counsel  and  mentor  only,  some  are  concerned  that  it  might  be  going  too  far  and 
could  shake  the  foundation  of  trust  in  360  that  the  Army  has  worked  so  hard  to  develop.  The 
Army  is  looking  to  see  how  participants  and  supervisors  react  to  this  new  approach.  It  is  cau¬ 
tiously  optimistic  that  supervisors  will  be  able  to  set  aside  the  information  from  evaluations  (as 
instructed)  and  that  raters  will  feel  comfortable  sharing  information  candidly.  But  only  time 
will  tell  whether  that  comfort  with  the  process  will  be  borne  out. 


Creating  a  Supportive  Organizational  Culture 

Many  360  experts  have  noted  that  the  success  of  360  is  partly  dependent  on  whether  raters 
and  ratees  accept  and  believe  in  the  value  of  the  process  (see,  for  example,  Bracken,  Timmreck, 
Fleenor,  and  Summers,  2001).  In  line  with  this  sentiment,  several  interviewees  discussed  the 
importance  of  cultural  change  management,  which  happens  slowly.  For  example,  in  the  Army, 

[MSAF  360]  didn’t  get  embraced  rapidly,  but  it  is  largely  part  of  the  process  now.  It  is  now 
more  accepted.  Now,  Command  360  will  also  go  through  that  process. 

Part  of  it  is  an  inculcation  piece.  It  took  [the  Army]  from  1997  to  2008  to  implement.  It’s 
a  cultural  change. 

Our  interviewees  expressed  very  different  levels  of  readiness  and  openness  to  a  360  pro¬ 
gram.  Some  services  have  had  very  limited  pilot  programs  ongoing  and  view  the  360  as  a  niche 
instrument  targeted  at  specific  populations  of  service  members.  These  individuals  were  reticent 
about  the  viability  of  a  broad-based  360  instrument  and  whether  such  a  tool  would  bring  about 
positive  or  negative  impact  to  the  service  overall: 

The  comfort  level  and  the  data  behind  the  value  to  the  culture  [are]  not  clear.  It  will  lead  to 
culture  erosion  problems,  e.g.,  bad-mouthing  or  anointing  each  other  as  kings  and  queens. 

If  it  is  done  with  280  [general  officers],  it  is  possible — but  for  70,000  officers,  it  is  harder  to 
communicate,  despite  resource  concerns.  It  could  be  useful  if  you  could  convince  them  to 
use  it  correctly,  but  it  is  difficult. 

Several  interview  participants  discussed  “a  culture  of  feedback.”  Generally  there  is  con¬ 
sensus  that  fostering  a  culture  of  personal  responsibility  and  bystander  intervention  is  valuable. 
However,  some  individuals  were  skeptical  whether  the  360  was  necessary,  sufficient,  or  even 
effective  at  creating  such  an  atmosphere: 

Now  it  is  a  core  of  changing  the  culture  where  feedback  is  common  and  it  is  not  uncommon 
to  say,  “Hey  you  need  to  check  that  language,  or  the  attitude.”  There  needs  to  be  a  mecha¬ 
nism  for  correcting  behavior.  A  coach  or  self-assessment  is  not  going  to  correct  the  behavior 
as  much  as  someone  grabbing  them  up  by  the  scruff  of  the  neck  and  making  the  point. 

I  think  we  could  improve  as  an  officer  corps  by  instituting  a  culture  of  feedback;  even 
though  it  is  uncomfortable,  it  could  be  helpful.  And  if  done  properly,  we  could  get  out  of 
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this  “How  do  you  help  bystanders  intervene?”  Feedback  is  another  form  of  bystander  inter¬ 
vention,  on  the  really  low  side.  A  culture  of  feedback  is  worthwhile,  but  360  is  not  a  single 
answer.  I  think  there  [is]  a  series  of  tools  that  we  could  use  that  would  be  worthwhile.  Not 
every  year,  not  all  grades  because  of  burn  out. 

The  services  have  different  levels  of  experience  with  a  process  like  the  360.  The  Army  is 
furthest  in  the  process  and  still  believes  that  it  has  not  yet  fully  integrated  360  into  the  culture. 
Many  of  the  other  services  were  worried  about  too  much  value  being  assigned  to  the  360,  and 
the  refrain  “not  a  silver  bullet”  was  often  heard.  Many  participants  did  not  feel  that  their  ser¬ 
vices  were  ready  for  a  culture  shock  as  big  as  360s  force-wide. 

Experts  also  point  to  cultural  readiness  as  an  important  factor  in  the  success  of  360 
efforts.  Brutus  and  Derayeh  (2002)  identified  challenges  to  implementing  360s,  including 
resistance  caused  by  a  lack  of  trust  in  the  system  (e.g.,  a  fear  of  being  identified  by  peers  or  a 
supervisor),  a  lack  of  strategic  alignment  with  organizational  objectives,  and  poorly  designed 
rating  instruments.  In  another  study,  managers  at  a  university  in  the  United  Kingdom  were 
concerned  about  a  lack  of  support  from  supervisors  for  developmental  activities  and  a  mis¬ 
match  between  the  competencies  addressed  in  the  360  program  and  those  emphasized  in 
other  organizational  policies,  such  as  criteria  for  promotion  (Mabey,  2001).  The  organizational 
environment  must  support  feedback-seeking  and  feedback-giving,  which  may  not  be  the  norm 
in  more-hierarchical  organizations  (Funderburg  and  Levy,  1997).  All  of  these  are  examples  of 
issues  that  might  make  large-scale  implementation  of  360s  in  the  services  particularly  difficult. 

Researchers  have  also  shown  that  leadership’s  attitudes  toward  feedback  processes  can 
influence  employees’  reactions  to  the  feedback  (see,  for  example,  Atwater  and  Brett,  2005). 
This  provides  further  support  for  the  sentiment  that  ensuring  buy-in  from  leaders  and  super¬ 
visors  in  the  services  before  implementing  360s  more  broadly  would  be  critical  to  ensuring 
success.  According  to  the  comments  from  our  interviews,  that  type  of  buy-in  in  the  services 
could  be  developed  gradually  over  time  (as  evidenced  by  the  Army)  but  would  take  years,  and 
forcing  360s  on  the  services  when  they  are  not  ready  for  them  could  lead  to  less  buy-in  rather 
than  more. 


Obtaining  Accurate  Ratings 

One  of  the  most  frequently  mentioned  concerns,  even  for  developmental  purposes,  was  related 
to  the  accuracy  of  the  information  that  would  be  obtained.  Concerns  included  potential  vari¬ 
ability  in  results  depending  on  which  individuals  participate  in  the  review,  how  to  evaluate  the 
quality  of  subordinate  input,  the  importance  of  contextualizing  results  within  an  individual’s 
chain  of  command,  and  the  need  to  vet  and  approve  the  360  questions  among  individuals 
who  will  participate  in  the  360  review.  Some  of  these  issues  have  also  been  the  focus  of  many 
empirical  studies. 

Our  interviewees  believe  that  honesty  is  a  big  concern,  particularly  when  using  360s  for 
evaluation,  and  experts  express  similar  concerns  (see  DeNisi  and  Kluger,  2000;  London,  2001). 
Participants  expressed  concerns  that  raters  might  whitewash  their  comments  for  a  variety  of 
reasons,  including  fear  of  reprisals,  friendship  with  the  ratee,  and  loyalty  to  the  ratee.  Others 
might  attempt  to  game  the  system  or  respond  quid  pro  quo.  Peers  might  be  motivated  to 
undercut  someone  they  see  as  competition,  and  subordinates  might  conspire  to  get  a  supervisor 
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they  dislike  fired,  particularly  if  their  own  jobs  were  in  jeopardy.  Still  others  might  not  take  the 
time  to  respond  carefully  or  accurately.  It  might  become  just  another  “check  the  box”  activity. 

The  consensus  of  our  interviewees  is  that  using  360s  in  evaluation  could  change  responses 
in  so  many  ways  that  the  results  could  ultimately  be  meaningless  and,  in  some  cases,  harmful. 
People  could  be  passed  over  for  promotion  because  of  false  information,  and  others  who  are 
toxic  might  still  slip  through  undetected.  In  the  words  of  one  of  our  participants, 

Conventional  wisdom  in  regards  to  360-degree  assessments  from  experts  and  researchers  is 
that  the  most  effective  use  of  360  assessments  is  to  enhance  professional,  individual  devel¬ 
opment.  Once  you  change  the  purpose  or  intent  of  a  360  from  development  to  evaluation, 
you  affect  the  willingness  of  raters  to  provide  candid  or  unfettered  feedback.  Raters  seem 
to  be  more  willing  to  provide  honest  feedback  if  the  ratings  remain  confidential  and  will  be 
used  only  by  the  person  rated  for  [his  or  her]  development.  When  promotions  are  at  stake 
and  results  may  be  distributed  to  others,  raters  tend  to  rate  more  positively. 

The  research  on  agreement  among  different  sources  of  raters  in  360  programs  and  whether 
ratings  are  consistent  and  stable  within  rater  sources  is  particularly  abundant.  For  example,  a 
meta-analysis  by  Viswesvaran,  Ones,  and  Schmidt  (1996)  found  somewhat  higher  interrater 
reliability  for  supervisors  than  for  peers.  Similarly,  a  meta-analysis  conducted  by  Conway  and 
Huffcutt  (1997)  found  that  reliability  was  highest  for  supervisors  and  lowest  for  subordinates, 
with  peers  in  the  middle.  In  general,  however,  the  reliability  coefficients  are  low  for  all  sources. 
A  number  of  studies  have  shown  that  reliability  is  also  influenced  by  characteristics  of  the  job, 
with  greater  agreement  among  sources  when  raters  have  more  opportunities  to  observe  ratees 
or  when  jobs  are  less  complex  or  have  more  concrete  or  observable  behaviors  (e.g.,  Harris  and 
Schaubroeck,  1988;  Heidemeier  and  Moser,  2009;  Warr  and  Bourne,  1999). 

Research  findings  on  agreement  between  different  sources  of  raters  is  mixed.  Meta¬ 
analyses  have  generally  found  low  correlations  for  self,  supervisor,  peer,  and  subordinate  rat¬ 
ings,  although  correlations  also  varied  among  different  pairs  of  sources,  with  higher  correlations 
between  peer  and  supervisor  ratings  than  between  other  pairs  of  rater  sources  (Conway  and 
Huffcutt,  1997;  Harris  and  Schaubroeck,  1988).  Conway  and  Huffcutt  (1997)  contended  that 
low  correlations  should  not  be  interpreted  to  mean  that  some  sources  are  inaccurate  or  should 
be  excluded;  instead,  the  authors  concluded  that  low  correlations  among  different  sources  dem¬ 
onstrate  that  the  different  sources  provide  unique  perspectives  on  performance. 

Likewise,  others  have  argued  that  ratings  from  different  sources  should  show  low  to  mod¬ 
erate  agreement;  high  levels  of  agreement  would  be  redundant  and  provide  little  benefit  given 
the  costs  of  collecting  360  feedback  (Borman,  1997;  Tornow,  1993).  Other  research  (e.g.,  Fac- 
teau  and  Craig,  2001;  Scullen,  Mount,  and  Judge,  2003;  Woehr,  Sheehan,  and  Bennett,  2005) 
has  shown  that  different  types  of  raters  have  similar  conceptualizations  of  performance,  sup¬ 
porting  the  notion  that  meaningful  comparisons  can  be  made  across  sources.  More  recently, 
LeBreton  and  colleagues  (2003)  raised  questions  about  the  use  of  correlations  to  assess  agree¬ 
ment  among  different  rater  sources  in  light  of  restriction  of  range  that  is  common  in  perfor¬ 
mance  ratings.  Using  an  alternative  approach,  LeBreton  and  colleagues  demonstrated  much 
higher  congruence  among  raters  than  previously  documented.1  Nevertheless,  it  is  possible  that 


1  The  authors  concluded  that  ratings  from  different  sources  may  provide  fewer  psychometric  benefits  than  previously 
thought  but  may  provide  psychosocial  benefits,  such  as  increased  job  satisfaction,  organizational  commitment,  and  percep- 
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ratings  from  some  sources  are  inherently  more  accurate  than  others.* 2  For  example,  Atkins  and 
Wood  (2002)  showed  that  self-ratings  can  sometimes  be  the  least  accurate  source  in  a  360.  And 
for  at  least  some  aspects  of  performance  in  the  military  (such  as  leadership  potential),  peers 
might  be  accurate  predictors  of  future  success.3 

Another  aspect  of  ratings  that  reflects  some  of  the  interviewees’  concerns  about  360s  is 
leniency.  In  their  meta-analysis  of  self  and  supervisor  ratings,  Heidemeier  and  Moser  (2009) 
found  that  leniency  was  moderated  by  the  purpose  of  the  ratings,  with  greater  leniency  in  rat¬ 
ings  conducted  for  administrative  and  developmental  purposes  than  in  ratings  conducted  for 
research.4  Surprisingly,  leniency  appeared  to  be  similar  in  ratings  conducted  for  developmental 
and  evaluation  purposes.  However,  a  variety  of  other  studies  show  that  raters  are  more  lenient 
when  ratings  have  a  direct  impact  on  ratee  outcomes — such  as  their  use  in  promotion  deci¬ 
sions.  In  a  study  of  undergraduate  project  teams,  Farh,  Cannella,  and  Bedeian  (1991)  found 
more  leniency  in  self  and  peer  ratings  when  ratings  were  used  to  determine  grades  rather  than 
for  developmental  purposes.  In  addition,  peer  ratings  in  the  developmental  condition  showed 
less  inflated  ratings,  better  discriminability,  and  higher  reliability.  In  research  on  performance 
evaluation  more  generally,  a  meta-analysis  showed  higher  ratings  for  administrative  decisions 
than  for  research  (Jawahar  and  Williams,  1997).  Longenecker,  Sims,  and  Gioia  (1987)  found 
that  executives  reported  inflating  performance  ratings  for  a  variety  of  political  motives,  includ¬ 
ing  reasons  related  to  employee  welfare,  such  as  job  benefits  (e.g.,  to  maximize  the  employee’s 
merit  increase)  or  career  opportunities  (e.g.,  to  avoid  creating  written  documentation  of  poor 
performance).  These  findings  support  arguments  favoring  360  feedback  for  developmental 
rather  than  evaluative  purposes.  When  it  came  to  our  interviewees,  participants  wholeheart¬ 
edly  agreed: 

My  concern  is  that  you  will  eliminate  the  efficacy  of  the  tool.  People  will  no  longer  be  frank 

and  honest. 


tions  of  organizational  justice.  Fleenor  and  colleagues  (2010)  noted  also  that  these  findings  suggest  that  self-ratings  should 
not  be  compared  to  separate  sources  of  ratings  from  others  but  to  combined  ratings  across  other  sources. 

2  The  accuracy  of  360  ratings  is  a  question  of  construct  validity.  Construct  validity  typically  is  established  by  analyzing 
correlations  among  variables  that  purport  to  measure  the  same  concept  or  phenomenon.  For  jobs  with  objective  measures 
of  performance,  one  may  be  reasonably  confident  in  the  construct  validity  of  360  ratings.  However,  because  “true”  perfor¬ 
mance  is  often  unknown — particularly  for  managerial  or  leadership  positions  for  which  there  are  typically  few  objective 
performance  criteria — the  construct  validity  of  360  ratings  has  been  assessed  by  correlations  with  other/external  measures 
of  performance.  However,  it  is  important  to  bear  in  mind  that  these  external  measures  may  themselves  be  subject  to  various 
confounds  or  other  factors  that  limit  their  reliability  or  validity. 

3  For  example,  a  study  of  U.S.  Army  officers  found  that  peer  ratings  of  potential  for  advancement  were  predictive  of  ratees’ 
promotion  to  general  (Downey,  Medland,  and  Yates,  1976). 

4  In  addition,  Heidemeier  and  Moser  (2009)  found  that  rating-scale  format  and  content  affected  leniency,  with  reduced 
leniency  when  social  comparison  scales  were  used,  when  ratings  focused  on  task  or  contextual  performance  rather  than  on 
traits  (such  as  personality  characteristics),  and  when  ratings  consisted  of  global  measures  rather  than  separate  dimensions  of 
performance.  {Task  performance  focuses  on  meeting  the  technical  requirement  of  one’s  job;  contextual  performance  addresses 
behaviors  contributing  to  the  broader  goals  of  the  organization,  such  as  helping  others  and  following  organizational  pro¬ 
cedures.)  Warr  and  Bourne  (1999)  also  showed  that  ratees  gave  themselves  higher  ratings  than  other  sources  did  on  more- 
subjective  criteria  (interpersonal  skills  and  flexibility)  and  lower  ratings  than  others  on  more-objective  criteria  (technical 
knowledge). 
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A  concern  with  rating  scales  is  that  scales  are  already  skewed.  Only  the  very  worst  perform¬ 
ers  get  anything  but  the  highest  score. 5 

There  is  probably  invalid  kind  of  stuff  on  both  ends  of  the  spectrum.  That’s  why  it  needs 
some  way  to  be  seen  overall  and  contextualized. 

You  have  to  watch  the  [phenomenon]  where  you  have  to  get  ten  subordinates  who  all  get 
together  and  all  collude  to  rate  someone  badly  especially  if  they  know  it  is  to  be  used  for 
promotion.  Especially  subordinates  who  have  an  axe  to  grind.  If  they  knew  that  the  360 
would  be  used  for  promotion,  they  could  use  it  wrongly.  Probably  we  would  say  no  to  evalu¬ 
ation  purposes  [for  this  reason]. 


Harming  the  Organization  When  Using  360s  for  Evaluation 

There  are  a  number  of  ways  that  360s  could  do  more  harm  than  good.  Resources  could  be 
wasted  on  programs  that  are  ineffective,  or  people  might  focus  their  performance  improvement 
efforts  on  areas  of  performance  that  an  organization  does  not  value.  However,  two  areas  of 
harm  are  of  greatest  concern  when  used  for  evaluation. 

Could  Ruin  the  Value  of  360  for  Development 

Several  participants  expressed  concern  that  the  trust  that  people  currently  have  in  360s  for 
development  could  easily  crumble  if  the  services  even  implied  they  might  be  considering  using 
them  for  evaluation.  The  Army  has  worked  for  years  to  develop  trust  from  personnel  that  the 
feedback  provided  to  others  will  be  anonymous  and  that  the  ratings  about  an  individual  will 
not  be  shared  with  anyone  else.  Interviewees  both  within  and  outside  the  Army  are  concerned 
that  it  would  take  many  years  to  rebuild  that  trust,  were  it  to  be  broken. 

Participants  also  mentioned  that  asking  participants  to  complete  two  separate  360s,  one 
for  evaluation  and  one  for  developmental  feedback,  would  not  be  a  viable  solution.  Doing 
so  would  be  costly  to  the  raters,  but  even  more  importantly,  participants  believed  that  raters 
would  easily  become  confused  and  eventually  they  would  approach  both  as  evaluation. 

This  is  consistent  with  concerns  expressed  by  many  experts  in  personnel  research.  For 
example,  in  an  article  on  how  to  help  improve  the  effectiveness  of  360  feedback,  DeNisi  and 
Kluger  (2000)  cautioned  that 

many  organizations  that  start  out  to  use  these  appraisals  only  for  developmental  purposes 
use  them  eventually  for  decision  making,  as  well.  In  fact,  we  are  familiar  with  several  orga¬ 
nizations  that  introduced  360-degree  appraisals  strictly  as  tools  for  development,  but  have 
either  begun  using  the  ratings  for  making  some  types  of  decisions,  or  they  are  considering 
doing  so.  To  do  so,  especially  after  stating  that  the  ratings  would  be  used  for  developmental 
purposes  only,  is  likely  to  hurt  an  organization’s  credibility  with  its  employees,  which  can 
lead  to  many  problems  and  reduce  employees’  willingness  to  change  their  behavior  follow¬ 
ing  feedback,  (p.  136) 


5  Note  that  performance  evaluations  in  the  military  have  long  suffered  from  just  this  concern.  Inflated  ratings  would  be 
even  more  problematic  with  360s  because  people  would  fear  retaliation  by  supervisors. 
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Could  Lead  to  an  Unwillingness  to  Discipline  Subordinates 

The  phrase  “popularity  contest”  arose  often  in  our  interviews.  For  some,  this  was  the  topic 
that  most  occupied  their  thinking  about  360  evaluations.  The  general  concern  is  that  360s  can 
reflect  how  much  a  subordinate  likes  his  or  her  superior  but  that  like  or  dislike  is  not  necessarily 
relevant  to  accomplishing  the  mission.  Some  are  worried  that  officers  may  concern  themselves 
with  how  they  are  perceived  and  may  be  less  likely  to  make  unpopular  choices  when  the  mis¬ 
sion  demands  it: 

You  might  be  getting  something  bad  too:  A  leader  putting  his  personal  ratings  above  the 
hard  decisions  he  has  to  make  to  get  the  mission  done. 

Oftentimes  the  commander  is  expected  to  come  in  and  right  the  ship,  and  in  doing  so  can 
fairly,  properly,  and  appropriately  make  some  tough  calls  or  shift  a  culture  on  a  given  base. 
[Commanders]  make  things  change  that  you  might  not  like.  So  to  some  degree,  the  360  is 
also  a  bit  of  a  popularity  contest. 

If  everybody  knows  it  is  a  development  tool,  it  will  lend  itself  to  honesty  and  to  people 
improving  themselves.  If  it  gets  into  the  performance  evaluations,  I  hope  it  also  doesn’t  lead 
to  people  trying  to  do  something  just  because  it  gets  good  scores. 

Additionally,  some  mentioned  that  360s  could  create  a  perception  (real  or  not)  that  dis¬ 
ciplinary  actions  are  taken  as  reprisal  for  negative  360  ratings.  Others  felt  that  they  could  lead 
to  excessive  investigations  and  ultimately  a  distrust  and  fear  of  the  process: 

There  is  no  recourse  for  a  [general  officer],  so  to  me,  what  we  are  doing  is  making  it  difficult 
for  leaders  to  discipline  soldiers  for  fear  of  being  investigated.  What  they  tell  you  is  that,  if 
you  haven’t  already  been  investigated,  it  will  happen  soon.  It  causes  a  lot  of  grief  for  families 
and  leaders  when  people  come  at  them  again  and  again. 

All  of  these  are  concerns  are  among  those  raised  by  experts  as  well  (see  London,  2001). 

Although  many  of  our  participants  raised  this  as  a  big  concern,  a  few  also  followed  up  by 
acknowledging  an  entirely  opposing  view.  They  explained  that  truly  good  leaders  might  still 
do  the  right  thing  and  make  the  tough  calls  in  spite  of  the  potential  for  their  actions  to  lead  to 
low  360  ratings: 

Most  commanders  don’t  want  to  be  disliked  but  want  to  make  the  right  decisions  because 
happy  troops  are  productive  troops.  If  we  have  guys  [who]  don’t  make  tough  choices,  then 
we  didn’t  pick  the  right  commander  to  start  with.  I  don’t  think  most  people  will  be  seduced 
by  this.  I  think  most  people  will  continue  to  do  what  they  think  is  right  and  the  hell  with 
what  happens. 

Most  of  those  who  offered  this  alternative  view  were  expressing  a  clear  uncertainty  about 
the  consequences  of  360s.  One  participant,  however,  did  not  see  it  as  an  uncertainty  at  all. 
They  instead  suggested  that,  although  this  is  often  presented  as  an  argument  for  not  imple¬ 
menting  360s,  leaders  in  the  military  simply  would  put  the  mission  first. 
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Making  360s  Logistically  Feasible  and  Conserving  Resources 

Our  interview  participants  were  concerned  about  the  feasibility  of  a  far-reaching  360  system. 
The  concerns  focused  largely  on  the  difficulty  of  managing  such  a  large  system,  the  challenges 
of  navigating  policy  to  support  such  a  program,  the  costs  of  implementation,  and  the  survey 
fatigue  that  service  members  may  feel  as  a  result  of  it. 

Representatives  from  some  services  mentioned  they  lack  the  capacity  to  scale  up  quickly. 
We  heard  concerns  about  data  management,  data  security,  storage,  and  responsibility  associ¬ 
ated  with  building  such  a  large  system.  Some  pointed  to  the  experience  of  the  Army  and  the 
rethinking  that  was  required  to  sustain  its  360  program  as  it  attempted  to  scale  the  system  up. 
Additionally,  there  were  concerns  that  a  service-wide  360  system  would  require  greater  cross¬ 
service  interoperability  than  is  currently  available.  Having  raters  outside  of  the  ratee’s  service 
would  likely  be  common,  but  the  current  information  technology  systems  do  not  connect  well 
across  services.  The  Joint  360  is  currently  grappling  with  exactly  this  challenge.  In  addition, 
some  noted  that  implementing  this  for  evaluation  would  require  significant  alterations  to  the 
existing  technological  systems  for  evaluation  to  accept  this  sort  of  new  input.  Such  a  change  to 
the  evaluation  systems  would  necessarily  have  to  happen  slowly. 

Coaching  was  another  logistics  and  cost  concern.  Finding,  training,  and  retaining  the 
required  number  of  coaches  needed  to  take  full  advantage  of  360s  would  be  costly.  Participants 
questioned  whether  it  would  be  logistically  feasible  or  cost-effective  to  provide  coaching  to  all 
officers  (see  the  next  section).  If  360s  were  administered  on  a  large  scale,  many  felt,  supervisors 
would  be  the  only  viable  option,  but  they  would  need  to  be  trained  before  they  could  take  on 
such  a  role;  however,  few  thought  that  sharing  the  information  with  even  trained  supervisors 
was  a  good  idea  (see  the  earlier  section  on  coaching  for  more  discussion  on  this).  The  Army’s 
solution  with  the  MSAF  is  to  offer  coaching  to  any  who  want  it.  It  has  found,  however,  that 
only  a  small  number  of  participants  take  up  that  offer,  thus  keeping  costs  down.  Given  that 
research  has  shown  that  the  utility  of  the  360  could  be  significantly  undermined  without  some 
sort  of  structured  follow-up  action,  some  have  asked  how  useful  it  is  to  administer  it  to  the 
large  majority  who  do  not  meet  with  coaches. 

Although  coaching  and  administration  costs  alone  might  be  a  deterrent  for  administer¬ 
ing  360s  force-wide,  raters’  time  costs  were  an  even  bigger  concern.  Our  participants  estimated 
that  it  takes  anywhere  from  ten  to  30  minutes  to  complete  the  various  360  rating  forms  they 
are  currently  using,  but  several  also  noted  that  it  usually  takes  significantly  longer  for  those 
who  add  written  comments  or  for  those  respondents  who  take  the  time  to  do  it  well.  Multiply 
that  by  the  number  of  people  above  and  below  each  service  member,  and  the  number  of  man¬ 
hours  spent  filling  out  forms  about  other  service  members  would  be  significant: 

We  are  always  conscious  of  making  sure  that  what  we  do  is  worth  the  [service  member’s] 
time  to  do  it.  We  tried  to  minimize  some  of  our  total  force  awareness  training  to  get  that 
into  a  smaller  window.  We  need  to  be  mindful  of  whatever  we  assign  takes  up  time. 

I  was  a  commander  and  I  had  seven  squadron  commanders  underneath  me,  so  I  [had]  to 
do  those  seven  because  I’m  their  supervisor.  There  are  ten  other  colonels  on  base,  so  I’ve  got 
to  do  them  because  I  am  their  peer.  And  then  my  bosses,  I  am  their  subordinate.  And  now 
you  are  looking  down  going,  “we  aren’t  going  to  get  any  work  done.” 
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If  I  am  a  commander,  I  am  going  to  be  asked  to  fill  out  surveys  on  every  one  of  my  officers 
because  I  am  their  supervisor.  Or  I  am  group  commander  or  wing  commander  and  it  cas¬ 
cades.  I  mean,  we  could  be  in  the  survey  business.  That  is  one  that  I  really  struggle  with,  is 
the  sheer  volume. 

Several  also  mentioned  that,  given  the  current  work  obligations  of  most  officers,  360s 
would  likely  have  to  be  completed  in  their  off  hours  at  home. 

Although  some  efficiency  could  be  achieved  by  not  having  everyone  complete  a  360  on 
everyone  else,  it  would  still  likely  be  burdensome.  As  the  Army  has  discovered,  when  raters 
are  chosen  by  the  ratee,  some  people  are  picked  as  raters  much  more  often  than  others.  As  a 
result,  the  numbers  of  ratings  those  people  have  to  complete  about  their  peers,  supervisors,  and 
subordinates  can  be  quite  large.  In  addition,  researchers  have  shown  that  a  minimum  number 
(at  least  six  or  seven  subordinates  and  four  to  seven  peers)  is  needed  to  achieve  good  reliability 
(Conway  and  Huffcutt,  1997),  and,  to  help  protect  anonymity,  more  is  better. 

Oversurveying  and  survey  fatigue  were  frequent  concerns,  particularly  because  of  the 
variance  in  unit  structures  within  and  across  the  services.  In  some  units,  an  individual  is  in 
command  of  six  officers;  in  others,  an  individual  is  in  command  of  over  100  officers.  The  obli¬ 
gation  to  fill  out  360s  on  that  many  individuals  would  be  overwhelming: 

I  haven’t  had  enough  input  from  raters  to  complete  my  survey.  ...  A  lot  of  the  individuals 
really  don’t  have  the  time  to  do  it. 

We  get  surveyed  to  death,  to  the  point  where  people  try  to  avoid  them.  You  only  fill  it  out 
when  they’ve  come  to  you  six  times.  This  will  just  be  more  of  that,  and  it  is  really  detrimen¬ 
tal  because  when  we  really  need  information  we  can’t  get  it  because  people  are  just  tired. 

Just  thinking  about  the  time  it  takes  to  fill  out  an  MSAF  .  .  .  just  multiply  that.  I  don’t 
know  if  that’s  a  fair  expectation  of  young  soldiers,  airman,  sailors,  and  Marines.  To  [take 
that  amount  of  time]  away  from  their  training,  away  from  their  duty  day. 

The  Army  is  already  seeing  the  impact  of  that  survey  fatigue,  with  many  participants 
unable  to  obtain  the  minimum  number  of  ratings  because  the  raters  are  unwilling  or  unable  to 
take  the  time  to  complete  them.6 

Additionally,  there  was  concern  about  the  frequency  of  360s.  If  the  program  were  linked 
to  evaluation,  then  the  360  feedback  would  have  to  be  gathered  and  processed  annually  before 
a  hard  deadline.  At  present,  the  Army  expects  officers  to  complete  an  MSAF  only  once  every 
three  years.  As  a  result,  expecting  it  to  be  completed  any  more  frequently  than  that  would 
likely  be  seen  as  highly  burdensome,  even  in  the  Army. 

That’s  another  piece — frequency.  How  often  do  we  do  this?  If  you  are  new  in  your  job,  do  I 
wait  a  year  before  I  give  it  to  you?  And  then  do  I  give  it  to  you  every  other  year?  Every  three 
years?  These  are  policy  questions  that  will  drive  cost  and  implementation. 

Similar  concerns  about  time  requirements  occur  in  civilian-sector  applications.  Brutus 
and  Derayeh  (2002),  for  example,  found  that  the  time  and  resources  required  to  implement 


6  See  the  earlier  discussion  on  this  in  Chapter  Four. 
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360-feedback  programs  was  a  major  deterrent.  Likewise,  in  post-360  surveys,  Edwards,  Ewen, 
and  Vendantam  (2001)  found  low  agreement  that  360s  were  an  efficient  use  of  time. 

A  few  noted  that  ultimately  it  was  a  question  of  prioritization  of  human  resources  over 
hardware,  and  the  trade-off  might  be  worth  it: 

For  [the  cost  of]  one  attack  helicopter,  the  Army  can  [pay  for  360s]  for  ten  years.  We  are 
way  underresourced  for  selecting,  promoting,  and  taking  care  of  our  people.  1  think  the 
cost,  if  the  program  is  properly  put  together,  is  justifiable. 

Others  felt  that  it  would  just  come  at  too  high  of  a  cost.  Some  other  aspect  of  people’s  time 
would  have  to  be  traded. 

They  also  expressed  concern  that  the  expense  along  with  workload  demands  would  lead 
to  cutting  corners.  A  need  to  make  the  rater  forms  short,  which  would  eliminate  much  of  the 
valuable  detail;  the  fact  that  people  would  be  overburdened  and  would  stop  putting  thought 
into  the  ratings  and  comments;  and  a  lack  of  meaningful  interpretation,  follow-up,  and  devel¬ 
opmental  assistance  were  some  of  cost-cutting  measures  they  foresaw — all  of  which  would 
negate  the  value  of  administering  it  in  the  first  place: 

1  just  think  logistically  .  .  .  once  we  start  to  try  to  implement  it,  it  would  almost  be  infea¬ 
sible.  So  then  you  would  force  us  to  go  “we  have  to  do  this.”  So  then  what  you  would  have 
to  do  is  cut  corners. 

If  you  want  it  quick,  you  are  going  to  get  it  quick.  And  if  you  get  it  quick,  then  are  we  filling 
the  square  that  we  did  a  360  and  missing  the  true  value. 

How  do  you  train  people  to  use  the  data  professionally?  Or  would  you  just  look  at  the  data 
and  go  “well,  it  looks  good  to  me;  no  one  really  complained  about  you”?  Versus  the  trained 
coach  [who]  sits  down  [and  spends  a  lot  of  time  going  over  it  with  the  ratee]. 


Including  Better  Measures  of  Success 

Measures  of  the  success  of  360s  in  the  services  are  fairly  limited.  Many  of  our  participants  had 
a  number  of  positive  things  to  say  about  how  useful  they  thought  their  own  experiences  receiv¬ 
ing  360  feedback  had  been.  Some  described  eye-opening  experiences  in  which  the  detailed 
comments  brought  to  their  attention  certain  aspects  of  their  behavior  they  had  never  previ¬ 
ously  realized  might  be  problematic.  A  few,  on  the  other  hand,  felt  that  the  results  were  not 
particularly  helpful  or  enlightening  because  they  did  not  learn  anything  new  that  they  did  not 
already  know.  Elowever,  reactions  to  360  feedback  can  be  positive,  even  when  there  is  no  mea¬ 
surable  improvement  in  performance. 

Perhaps  the  most  important  question  regarding  the  use  of  360-degree  feedback  is  whether 
individual  and  organizational  performance  improves  as  a  result  of  this  technique  and  under 
what  circumstances.  So  do  360s  in  the  military  lead  to  performance  improvements?  That  ques¬ 
tion  remains  largely  unanswered.  Discussions  with  DoD  stakeholders  suggest  that  most  360 
efforts  currently  in  place  do  not  have  any  existing  method  for  evaluating  this.  Some  (such  as 
the  MSAF)  collect  satisfaction  ratings  from  the  person  who  was  rated  as  their  primary  outcome 
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measure.  Others,  however,  have  no  outcome  measures  in  place  at  all.  Many  agreed,  however, 
that  such  measures  would  be  useful: 

Okay,  a  year  later,  six  months  later,  you  worked  up  a  performance  plan  with  a  coach.  Did 
you  ever  implement  that?  Do  you  have  examples  in  the  past  year  as  a  supervisor,  leader, 
commander  that  you  practiced  what  was  identified  in  your  360?  I  don’t  have  [those]  data, 
but  [they]  would  be  interesting. 

One  of  the  problems  in  determining  whether  individual  performance  improves  is  that  it 
would  require  a  systematic  data-collection  effort.  And  taking  the  time  to  do  that  is  just  one 
more  burden  to  add  to  the  360  process.  Nevertheless,  when  we  asked  whether  the  services 
had  considered  how  to  measure  success,  many  agreed  that  such  measures  are  needed  and  are 
important  in  evaluating  whether  360s  are  worth  the  cost: 

Why  360s?  How  will  they  make  us  better  [individually,  collectively]?  Supporting  evidence 
inside  or  outside  DoD?  There  is  an  assumption  in  some  circles  that  it  can  be  used  to  root 
out  toxic  leaders — what  is  the  effect  on  the  climate?  .  .  .  For  development,  what  return  on 
investment  is  the  Army  receiving? 

In  one  discussion,  the  idea  was  raised  of  taking  360s  at  regular  intervals  during  the  course 
of  a  career.  This  longitudinal  approach  would  provide  service  members  an  opportunity  to  see 
if  they  have  improved  in  certain  areas  over  time.  For  example, 

if  every  three  years  I  take  a  360,  and  I  can  see  that  I  consistently  take  a  hit  in  whatever. 
Maybe  I  am  consistently  very  fair  but  horrible  communicator.  That’s  hugely  valuable.  I 
would  rather  have  [those]  data  than  water  [them]  down  with  ratings.  That  is  a  huge  oppor¬ 
tunity,  but  logistically  that  is  challenging. 

However,  in  discussing  whether  360s  could  be  used  to  compare  someone’s  ratings  over 
time  to  see  whether  there  was  improvement,  some  participants  agreed  that  it  might  be  useful 
but  noted  that  such  a  longitudinal  look  was  not  currently  possible: 

The  reports  do  not  provide  a  comparison  tool  [for  results  over  time].  A  longitudinal  look 
would  be  very  valuable.  Also,  because  it’s  so  recent,  it’s  going  to  take  few  years  before  some¬ 
one  can  see  the  change. 

But  even  this  longitudinal  approach  has  shortcomings  because  only  the  individual  can  see 
the  results,  so  it  is  difficult  at  an  organizational  level  to  know  whether  individuals  are  improv¬ 
ing  as  a  result  of  the  use  of  360s  or  not — and  to,  in  turn,  determine  the  benefit  to  the  organiza¬ 
tion.  In  addition,  other  participants  pointed  out  that  people  change  positions  so  frequently  in 
the  military  that  differences  from  one  360  administration  to  the  next  might  just  simply  reflect 
a  change  in  the  people  providing  the  ratings  or  new  expectations  in  a  completely  new  job.  As 
a  result,  a  longitudinal  look  might  prove  inconclusive. 

Finally,  one  interviewee  suggested  using  unit  360s  or  command  climate  surveys  as  a  tool 
for  measuring  behavioral  improvement. 

I  think  that  is  best  handled  by  a  climate  survey.  .  .  .  Most  senior  commanders  get  briefings 
periodically  on  numbers  of  people,  money  left  in  the  bank,  how  many  tanks  are  running, 
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how  many  aircraft  are  down  for  various  things.  But  only  irregularly  do  we  get  the  percep¬ 
tion  of  organizational  readiness  of  a  unit.  ...  If  I  were  king,  I  would  say  start  out  either 
simultaneously  with  or  even  ahead  of  this  individual  360  and  do  a  360  on  your  discernible 
organizations,  my  hope  is  that  it  would  be  done  at  the  same  time.  I  think  the  best  way  to 
measure  the  collective  impact  of  leader  development  efforts  is  by  looking  at  how  they  play 
out  in  the  real  world,  and  the  real  world  can  be  assessed  rather  simply  and  not  very  expen¬ 
sively  by  periodic  climate  surveys. 


Tailoring  360s  to  the  Services'  Needs 

The  need  for  service-level  autonomy  over  implementation  decisions  was  mentioned  during  sev¬ 
eral  interviews  and  from  respondents  from  all  organizations: 

[Human  resource]  systems  need  to  be  tailored  to  each  of  the  services.  The  services  have 
different  cultures  and  they  are  different  for  a  reason.  It  may  match  one  but  be  a  [forced]  fit 
for  others.  The  Army  may  want  someone  who  follows  commands,  while  the  Air  Force  may 
want  a  technical  specialist  who  thinks  outside  the  box  and  can  think  of  a  number  of  ways 
to  break  into  a  database. 

Participants  also  noted  that,  even  within  a  service,  a  one-size-fits-all  approach  will  not 
work  well.  Who  participates  as  raters  and  whether  it  makes  sense  for  some  people  to  even 
receive  360s  are  important  questions  that  should  not  be  overlooked: 

I  think  you  need  to  take  into  account  some  career  field  differences.  You  won’t  be  able  to  ask 
about  leadership  for  some  of  the  occupations  and  younger  folks. 

When  you  have  [an]  Army  or  Marine  [Corps]  construct,  where  the  organization  is  a  lot 
flatter,  and  you  got  platoons,  and  companies,  and  battalions,  there  are  more  leadership 
opportunities,  more  opportunities  to  manage  people. 

They  also  discussed  a  need  for  flexibility  in  other  key  design,  administration,  and  tech¬ 
nological  features,  such  as  the  opportunity  to  customize  questions  to  occupations  and  grades: 

For  it  to  be  effective,  the  officer  has  to  be  in  a  leadership  environment.  I  don’t  know  how 
you  can  ask  the  same  kinds  of  questions.  Maybe  you  could  have  a  tailored  piece  for  different 
ranks.  .  .  .  Even  in  operational  areas,  a  young  fighter  pilot  trying  to  get  upgraded  to  instruc¬ 
tor  pilot,  he  is  flying,  he  is  supposed  to  be  in  mission  study,  he  is  supposed  to  be  flying  as 
much  as  he  can;  how  do  you  assess  his  leadership?  He  is  a  little  different  than  a  section  com¬ 
mander  in  a  thousand-person  maintenance  squadron. 

In  addition,  the  services  need  to  have  enough  flexibility  to  decide  who  was  in  a  position 
to  benefit  most  from  a  360.  There  was  consensus  among  the  majority  of  our  participants  that 
360s  would  be  most  useful  and  much  more  cost-effective  if  they  were  implemented  only  for 
personnel  in  higher-level  grades  and  positions: 

I  would  think  you  want  to  hold  off  until  field  grade  0-4-0-5.  This  is  really  because  of  the 
time  involved.  The  overhead  is  too  high  time-wise. 
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Development  should  start  much  lower  level;  feedback  for  selection  should  start  at  0-6  and 
0-7. 


I  think  360  for  development  is  a  wonderful  thing,  but  we  can’t  do  it  for  everybody  and 
should  focus  on  the  midgrade  officer  and  higher.  Captain  is  good  place  to  start. 

Money,  time,  fatigue.  Personally  I  don’t  think  it’s  worthwhile.  All  ranks  need  development, 
just  like  a  general  officer.  .  .  .  [But]  the  feedback  [gotten]  from  the  population  (particularly 
subordinate  and  peers)  won’t  be  as  helpful  for  junior  people.  The  senior  look  is  helpful  for 
anyone. 

From  0-2  or  0-3,  we  have  the  problem  of  an  unstable  subordinate  population — they  move 
around  too  fast — and  the  relationship  could  be  too  close  here  to  be  done  effectively  without 
serious  professional  guidance. 

Finally,  some  of  our  participants  commented  that,  while  officers  are  certainly  an  impor¬ 
tant  group  to  include  in  360  efforts,  they  are  not  the  only  relevant  group.  It  would  be  useful  to 
consider  including  enlisted  personnel  and  civilians,  but  when  to  incorporate  them  and  how  to 
do  so  should  again  be  left  up  to  the  services’  discretion. 


CHAPTER  SEVEN 


Implications  for  the  Military  Services 


When  360  programs  are  implemented,  they  should  have  a  specific  purpose  in  mind  that  aligns 
with  organizational  goals.  Without  a  well-understood  purpose,  it  is  not  possible  to  know 
whether  the  system  is  working  and  is  worth  the  time  and  resources  that  an  organization  must 
invest.  This  applies  to  private-sector  entities  as  well  as  the  military.  So,  the  first  step  in  intro¬ 
ducing  or  expanding  use  of  360s  is  to  determine  its  purpose. 

Currently,  360s  in  the  military  services  are  used  for  developmental  purposes  only,  not 
as  an  evaluation  of  past  performance.  The  overall  consensus  from  our  interviews  is  that  the 
primary  purpose  of  360s  should  continue  to  be  for  development,  particularly  for  officers  in 
leadership  positions.  In  this  context,  results  are  intended  to  help  an  officer  hone  his  or  her  skills 
as  an  effective  leader  by  giving  an  individual  the  opportunity  to  develop  goals  and  courses  of 
action.  In  this  chapter,  we  conclude  with  our  assessment  of  whether  360s  are  the  right  tool  for 
the  U.S.  military. 


Purpose  and  Design  Matter 

According  to  Bracken,  Timmreck,  Fleenor,  and  Summers  (2001), 

Multi-source  feedback  looks  deceptively  easy.  It  is  like  watching  a  proficient  athlete  and 
saying  “I  could  do  that”  without  any  knowledge  of  the  training  and  preparation  required 
for  excellence.  In  much  the  same  way,  an  effective  360  feedback  system  that  creates  sustain¬ 
able,  focused  behavior  change  requires  both  good  tools  and  good  implementation.  Systems 
that  fall  short  in  one  or  more  of  the  [key  features]  will  probably  not  “hurt”  anyone,  and  may 
actually  survive  in  a  benign  way.  The  problem  is  that  they  will  not  create  behavior  change, 
at  least  in  the  manner  envisioned  by  the  organization.  At  best  they  are  a  waste  of  money. 

At  worst  they  divert  time  and  resources  toward  activities  that  will  not  bear  fruit  and  may 
cause  employee  morale  problems. 

Perhaps  the  most  important  step  in  adopting  or  expanding  the  use  of  360  feedback — 
in  the  military  or  elsewhere — is  to  determine  and  provide  guidance  on  the  purpose  of  the 
program.  In  the  military,  for  example,  these  assessments  have  been  used  primarily  for  force 
development — to  help  build  better  leaders  and  help  individuals  understand  gaps  in  how  they 
perform  their  jobs.  Congress  has  asked  whether  there  is  a  role  for  360  feedback  in  the  appraisal 
system.  If  used  for  evaluation,  what  would  be  the  purpose?  Some  interviewees  speculated  that 
Congress’s  intent  for  including  360  in  evaluations  is  “to  find  the  toxic  leaders  or  the  toxic  lead¬ 
ership  characteristics  that  we  might  not  know  otherwise.” 
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But  as  we  heard  during  our  interviews,  Congress’s  reasoning  for  implementing  360  for 
all  officers  in  the  military  services  has  neither  been  clearly  expressed  nor  is  it  well-understood: 
“What  are  we  trying  to  fix  here?”  “I’ve  yet  to  see  someone  who  comes  and,  through  a  military 
lens,  says,  ‘we  are  going  to  do  360s,  here  are  our  objectives,  here  is  our  plan.’”  If  the  services 
were  to  implement  360s  force-wide,  they  would  need  a  statement  of  goals  that  clearly  articu¬ 
lates  what  the  program  is  meant  to  accomplish — something  more  specific  than  “get  better 
leaders”  or  “screen  toxics.” 

Specificity  of  goals  is  important  because  views  diverge  on  the  utility  of  360,  as  the  discus¬ 
sion  in  the  previous  chapters  illustrates.  Some  believe  that  existing  tools  can  accomplish  broad 
goals,  such  as  “better  leadership,”  in  a  more  convenient  and  cost-effective  way: 

I  think  there  is  a  convenient  and  powerful  role  for  organizational  climate  surveys  that  take 
care  of  a  lot  of  this,  and  [they  are]  so  much  easier  to  do  in  the  short  term.  How  do  you  mea¬ 
sure  progress?  It  is  going  to  take  a  while,  but,  by  conducting  climate  surveys  over  time,  you 
see  trends,  and  instantly  you  can  get  a  good  feel  for  what  is  good  and  bad.  Climate  surveys 
are  much  less  threatening  to  a  commander  than  a  subordinate  assessment.  Climate  assess¬ 
ment  can  provide  a  lot  of  useful  information  for  discussion  for  how  things  are  going  in  the 
organization  that  can  have  a  side  impact  of  quality  of  leadership  and  so  forth. 

Others  feel  that  360s  can  make  a  contribution  as  part  of  a  suite  of  tools: 

But  we  also  think  that  there  is  a  way  to  use  360,  command  climate  surveys,  and  other  tools 
to  understand  the  impact  of  the  commander  on  [his  or  her]  unit.  The  360  is  one  tool  to  use 
to  understand  how  leaders  are  doing  in  command.  But  we  also  think  [that]  these  tools  are 
useful  to  develop  people  before  they  get  to  command. 

Regardless  of  how  the  services  decide  to  use  360s — and  we  anticipate  that  the  goals  and 
use  could  and  probably  should  differ  by  service — a  clear  statement  of  goals  is  an  essential  start¬ 
ing  point.  Without  such  clear  statements,  it  will  be  impossible  to  evaluate  whether  the  program 
is  designed  properly,  working  as  intended,  or  even  needed  at  all. 


Recommendations  for  Using  360s  in  the  Military 

"No"  for  Evaluation 

Based  on  our  research  on  360s,  both  within  and  outside  a  military  setting,  we  advise 
against  incorporating  360s  in  the  officer  evaluation  system  at  this  time.  Although  the 
results  of  our  interviews  suggest  that  implementing  360s  force-wide  for  evaluation  purposes 
would  technically  be  feasible — that  is,  systems  could  be  changed,  policies  could  be  revised,  new 
guidance  could  be  issued,  and  personnel  could  be  trained  to  handle  the  use  of  360  feedback 
information  in  military  performance  evaluations — it  simply  would  not  be  advisable  for  several 
reasons. 

For  one,  it  could  ruin  developmental  applications,  and  the  Army’s  experience  shows  how 
long  it  can  take  to  develop  trust  in  such  a  process.  Trying  to  implement  two  separate  systems 
could  be  confusing  to  raters  and  increase  survey  burden  on  the  force.  Using  360s  as  part  of 
evaluation  processes  could  also  create  perverse  incentives  for  commanders  who,  thinking  about 
the  impact  of  360  feedback  on  their  performance  appraisals,  could  make  decisions  (consciously 
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or  unconsciously)  based  on  a  desire  to  be  liked  by  the  troops  that  might  be  inconsistent  with 
the  mission.  There  are  similar  concerns  about  incentives  for  raters  who  could  sabotage  reviews 
in  an  attempt  to  get  rid  of  an  unpopular  leader  or  could  whitewash  ratings  for  friends  or  for 
fear  of  retaliation. 

Moreover,  there  are  a  host  of  logistical  issues  that  are  unresolved,  perhaps  the  most  impor¬ 
tant  of  which  is  the  need  to  develop  formal  guidance  in  the  use  of  the  tool.  Because  of  the 
complexity  of  using  360s  in  the  evaluation  system,  mandating  their  use  could  lead  to  largely 
useless  or  meaningless  ratings  that  are  difficult  to  interpret  without  considerable  context.  The 
performance  evaluation  system  fundamentally  provides  a  record  of  past  performance.  It  also 
signals  how  well  the  officer’s  performance  aligns  with  the  traits  that  are  valued  by  the  service. 
Should  360s  become  part  of  the  formal  evaluation  system,  clear  guidance  on  how  to  treat  the 
results  would  be  required  when  there  are  differences  among  superior,  peer,  and  subordinate 
assessments. 

Perhaps  the  key  concern  in  using  360s  beyond  developmental  purposes  is  the  potential 
impact  on  selection  boards  and  the  promotion  process.  For  the  purposes  of  determining  the 
effect  of  expanding  performance  evaluation  reports  to  include  a  360-degree  assessment,  the  pro¬ 
vision  of  law  that  would  directly  link  the  assessment  to  selection  boards  and  the  promotion  pro¬ 
cess  is  Section  615  of  Title  10  of  the  United  States  Code.  While  a  requirement  to  include  360s  in 
performance  evaluations  may  not  reference  using  that  information  for  promotion  consideration 
purposes,  an  officer’s  evaluation  reports  are  one  of  the  primary  documents  reviewed  by  board 
members.  Thus  if  the  results  of  360s  become  a  required  element  of  evaluation  reports,  that  infor¬ 
mation  would  be  provided  to  selection  boards,  absent  a  change  in  service  policy  as  to  the  final 
disposition  of  performance  reports,  and  would  be  used  during  the  board’s  deliberations. 

The  inclusion  of  this  information  in  board  deliberations  is  of  concern  for  several  reasons. 
The  information  from  raters  is  usually  anonymous  and  therefore  cannot  be  challenged  by  the 
ratee.  Information  in  360s  can  sometimes  be  inaccurate  because  of  rater  lack  of  care  or  skill 
in  providing  ratings.  In  a  high-stakes  situation,  such  as  promotions,  raters  could  be  dishon¬ 
est  in  attempts  to  positively  or  negatively  impact  board  selection  decisions  with  no  potential 
for  recourse.  Finally,  360  ratings  often  require  an  understanding  of  the  context  in  which  the 
ratings  were  provided.  A  difficult  assignment  might  lead  to  low  ratings  despite  exceptional 
performance  on  the  part  of  the  ratee.  Such  context  would  not  be  obvious  to  the  board  and  has 
the  potential  to  worsen  rather  than  improve  the  accuracy  and  fairness  of  promotion  decisions. 
Such  information  should  be  included  in  board  deliberations  only  after  careful  examination  of 
its  accuracy,  both  initially  and  over  time. 

We  acknowledge  that  there  may  still  be  a  desire  for  new  avenues  by  which  to  incorporate 
input  from  peers  and  subordinates  into  the  performance  evaluation  system.  Though  we  did  not 
explore  specific  options  in  this  study,  other  approaches  to  doing  so  could  be  worth  considering. 
For  example,  the  evaluation  rating  form  itself  could  be  redesigned  so  that  raters  are  required 
to  comment  on  what  peers  and  subordinates  think  about  the  person  being  rated — formal¬ 
izing  the  informal  inputs  that  many  supervisors  already  gather  from  peers  and  subordinates. 
Encouraging  this  type  of  information-sharing  and  training  raters  on  how  to  solicit  and  use 
such  information  is  one  way  for  the  performance  evaluation  system  to  capture  a  broader  range 
of  perspectives.  Although  this  type  of  alternative  may  be  worthwhile,  we  cannot  make  a  spe¬ 
cific  recommendation  because  the  aim  of  this  study  was  to  determine  whether  including  360s 
in  performance  evaluations  would  be  advisable.  We  do  not  believe  that  360  assessments  are  the 
appropriate  tool  for  this  purpose — at  least  not  at  this  time. 
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"Yes"  for  Development,  but  Proceed  with  Care 

Based  on  the  literature  and  the  interview  findings,  we  would  advise  the  use  of  360s  for 
development  purposes.  360s  could  be  used  for  leader  development  and  made  available  as  a 
service  to  individuals  hoping  to  improve.  The  tool  could  be  offered  to  anyone  who  wants  to  use 
it,  along  with  coaching  to  help  service  members  evaluate  the  results  and  incorporate  them  into 
self-improvement  goals.  360s  could  also  be  adapted  for  special  uses,  such  as  in  training  courses. 

In  addition,  360s  could  be  used  to  provide  an  aggregate  view  of  leadership  across  the 
force — something  that  current  tools  are  not  well  positioned  to  provide.  Individuals  could  com¬ 
pare  their  performance  with  that  of  other  service  members.  Leaders  could  use  the  360  to  iden¬ 
tify  groups  (such  as  units,  occupations,  or  bases)  in  which  ratings  are  consistently  better  or 
worse  than  others  (regardless  of  the  person  in  charge).  The  feedback  could  be  used  to  identify 
areas  that  could  benefit  from  developmental  assistance  or  other  support  interventions.  The 
results  could  also  be  used  to  identify  force-wide  strengths  and  weaknesses  in  performance. 
For  these  purposes,  the  tool  could  be  implemented  for  a  sample  of  the  force,  not  as  a  census,  if 
survey  fatigue  is  a  concern. 

Such  an  aggregate  use  of  360  information  was  mentioned  briefly  by  only  a  few  of  our 
participants.  Nevertheless,  this  application  is  in  keeping  with  the  well-supported  goal  of  using 
360  feedback  for  developmental  purposes.  To  the  extent  that  360s  could  be  used  to  supplement 
or  complement  existing  or  future  climate  assessments,  the  idea  has  merit.  Yet  care  should  be 
taken  to  ensure  that  any  overlap  with  existing  climate  assessments  is  intentional.  Otherwise, 
the  effort  could  be  duplicative  of  efforts  already  under  way  and  hence  a  waste  of  raters’  time. 

The  bottom  line  is  that  making  360  feedback  available  for  developmental  use  in  the  mili¬ 
tary  services  is  a  good  idea  and  is  essentially  how  the  tool  is  being  used  today.  Each  of  the  ser¬ 
vices  uses  360s  in  ways  that  it  finds  useful  in  support  of  its  personnel  management  goals  and  in 
line  with  service  culture.  While  use  of  360s  could  be  expanded  in  the  services,  how  and  when 
that  is  done  should  be  determined  by  the  services  themselves.  On  the  other  hand,  OSD  could 
consider  using  a  similar  tool  to  solicit  input  at  an  aggregate  level  to  survey  personnel  leadership 
skills  across  the  services,  similar  to  the  use  of  climate  surveys  today. 

A  mandate  to  use  360s  department-wide  could  waste  significant  resources.  A  one-size- 
fits-all  approach  would  be  ineffective  in  most  cases  because  the  needs  of  the  services,  of  individ¬ 
ual  service  members,  of  particular  career  fields,  and  even  individual  positions  differ.  Moreover, 
other  tools  are  in  place  within  the  services  that  might  be  achieving  similar  goals.  So  further 
implementation  needs  to  consider  efforts  already  under  way  and  what  360s  would  add.  And 
although  cost  of  the  tool  itself  is  relatively  low,  time  costs  for  personnel  (particularly  to  get 
maximum  benefit  of  360s)  is  high,  so  the  purpose  of  implementing  the  program  and  how  it 
aligns  with  organizational  goals  need  to  be  made  clear.1 

Our  assessment  suggests  that  many  features  of  the  Army  MSAF  program  (such  as  service- 
specific  competencies  and  rater  qualifications)  would  likely  need  to  be  tailored  for  implemen¬ 
tation  in  other  services,  even  though  it  is  the  most  developed  among  the  service  programs 
and  a  useful  model  for  how  360s  could  be  used.  Although  the  Center  for  Army  Leadership 


1  Participants  with  experience  administering  360s  in  the  services  cited  times  to  complete  forms  as  ranging  from  ten 
to  30  minutes.  However,  they  noted  that,  for  those  who  provide  comments,  the  forms  can  take  much  longer  to  com¬ 
plete.  Time  costs  for  coaching  and  follow-up  after  receiving  the  results  were  not  estimated,  but  those  who  had  experience 
receiving  intensive  coaching  in  a  developmental  setting  spread  over  days  or  weeks  described  that  scenario  as  the  ideal  but 
time-consuming. 
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has  offered  to  adapt  the  MSAF  and  provide  additional  support  to  address  any  service-specific 
needs,  the  services  also  might  want  to  develop  their  own  tools  and  in-house  support  staff. 

Nevertheless,  as  shown  by  the  Army’s  existing  MSAF  effort,  providing  the  needed  infra¬ 
structure  to  support  the  tool  is  no  small  undertaking.  An  experienced  staff  is  needed  to  develop 
and  maintain  the  survey  content,  supporting  materials,  and  technological  interface;  review 
survey  responses  and  requests  for  data;  provide  coaching  and  follow-up;  evaluate  success  of  the 
tool;  and  more.  Some  of  the  services  already  have  existing  small-scale  efforts  that  are  supported 
by  either  external  vendors  or  in-house  staff.  Those  efforts  could  be  expanded,  although  changes 
would  be  needed  to  adapt  them  to  other  populations  or  uses,  requiring  larger  in-house  staff  or 
further  negotiation  with  vendors.  Regardless,  the  Army’s  existing  tools  are  available  to  anyone 
with  a  CAC  who  wants  to  use  them.  This  means  that  anyone  in  the  military  can  voluntarily 
log  in  and  initiate  a  360  on  himself  or  herself.  As  a  result,  the  Army’s  tool  could  be  adopted  by 
anyone  in  the  military  interested  in  using  it  off  the  shelf. 

Current  Implementation  Is  on  the  Right  Track 

Overall,  our  interviews  showed  that  the  spirit  of  360s  clearly  resonates  with  the  services. 

The  services  clearly  value  good  leadership  behaviors  and  tools  that  can  help  develop  good 
leaders,  and  the  360  is  one  tool  that  has  value  in  developing  leaders.  Interviewees  agreed  that 
achieving  the  mission  at  the  expense  of  people  and  values  is  not  acceptable.  The  services  want 
to  eradicate  problematic  leadership  behaviors,  such  as  toxic  leadership  (poor  treatment  of  sub¬ 
ordinates)  and  unethical  behavior  (fraternizing,  sexual  harassment,  and  assault).  But  there  is 
a  general  consensus  that  the  360  is  not  the  right  tool  for  these  specific  purposes — other  tools 
exist  that  can  be  used  more  effectively  with  fewer  burdens  on  the  force. 

Stakeholders  generally  find  360  feedback  beneficial  but  recognize  that  360s  are  not  a 
silver  bullet.  Experts  in  the  held  of  personnel  research  agree  and  caution  that  overzealous  use 
of  360s  without  careful  attention  to  the  content,  design,  and  delivery  can  even  be  harmful  to 
an  organization  and  not  worth  the  time  and  money  spent  on  them.  For  example,  according  to 
DeNisi  and  Kluger  (2000), 

the  design  of  any  feedback  intervention  requires  thought  and  consideration  of  the  factors 
that  are  likely  to  make  it  more  effective.  Without  such  consideration,  organizations  might 
actually  be  designing  systems  that  are  not  cost-effective  in  the  long  run.  (p.  138) 

Each  of  the  services  has  used  360  assessments  to  some  degree — with  the  Army  having 
the  most  experience  to  date.  The  services  have  commissioned  many  reviews  on  this  topic,  and 
many  service  members  have  chosen  360  feedback  as  their  thesis  topics  at  the  various  command 
and  staff  colleges.2  Nevertheless,  while  they  agree  that  the  tool  can  be  effective  in  some  circum¬ 
stances,  the  services  also  see  other  ways  to  solicit  input  from  subordinates  on  leadership  behav¬ 
ior.  Moreover,  they  believe  that  mandating  360s  force-wide,  even  for  development  purposes, 
is  not  necessarily  the  right  answer  to  solving  leadership  problems  within  the  services.  Service 
cultures  may  not  be  equally  supportive  of  360s,  so  mandating  their  use  could  lead  to  failure. 
Rather,  it  is  more  advisable  to  allow  the  services  to  continue  on  their  current  paths,  expanding 
the  use  of  360s  in  a  way  that  is  tailored  to  individual  service  needs  and  goals. 


2  For  some  examples,  see  Fiscus,  2011;  Fitzgerald,  1999;  Hancock,  1999;  Leboeuf,  1997;  Nystrom,  2001;  Pointer,  1996; 
Psotka,  Legree,  and  Gray,  2007;  Reese,  2002;  and  Williams,  2005. 


Abbreviations 


360 

360-degree  assessment 

CAC 

common  access  card 

CCL 

Center  for  Creative  Leadership 

DoD 

Department  of  Defense 

GS 

General  Schedule  pay  scale 

MSAF 

Multi-Source  Assessment  and  Feedback 

NDAA 

National  Defense  Authorization  Act 

OSD 

Office  of  the  Secretary  of  Defense 

OUSD/P&R 

Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness 
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